Jul 06 13:37:34.934069 ip-10-0-134-202 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 13:37:34.934080 ip-10-0-134-202 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 13:37:34.934087 ip-10-0-134-202 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 13:37:34.934306 ip-10-0-134-202 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 13:37:45.109936 ip-10-0-134-202 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 13:37:45.109951 ip-10-0-134-202 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a06ebacd2b7749849006e4501883bb39 -- Jul 06 13:39:44.140539 ip-10-0-134-202 systemd[1]: Starting Kubernetes Kubelet... Jul 06 13:39:44.522950 ip-10-0-134-202 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 13:39:44.522950 ip-10-0-134-202 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 13:39:44.522950 ip-10-0-134-202 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 13:39:44.522950 ip-10-0-134-202 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 13:39:44.522950 ip-10-0-134-202 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 13:39:44.522950 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.520228 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525540 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525554 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525558 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525561 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525564 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525567 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525570 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525572 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525577 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525579 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525582 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525584 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525587 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525589 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525592 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525595 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525598 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525602 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525606 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:39:44.527171 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525609 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525612 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525616 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525618 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525622 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525625 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525628 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525630 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525633 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525635 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525638 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525641 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525644 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525646 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525649 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525651 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525654 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525656 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525659 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525661 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:39:44.633832 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525664 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525668 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525670 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525673 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525675 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525677 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525680 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525682 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525684 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525687 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525691 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525694 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525697 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525701 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525704 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525707 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525710 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525715 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525717 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:39:44.636379 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525720 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525722 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525725 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525728 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525731 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525733 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525737 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525740 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525742 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525745 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525748 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525750 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525753 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525758 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525762 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525765 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525767 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525770 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525773 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:39:44.637253 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525776 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525778 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525781 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525797 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525810 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525813 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525816 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525819 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525822 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525825 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525827 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525830 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525833 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525835 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525838 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525840 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525843 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525846 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525848 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525851 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:39:44.638625 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525853 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.525856 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528343 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528352 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528356 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528359 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528362 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528365 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528368 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528370 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528373 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528377 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528380 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528382 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528385 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528388 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528390 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528393 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528396 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528400 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:39:44.639494 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528403 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528406 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528409 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528412 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528415 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528418 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528423 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528427 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528430 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528433 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528436 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528439 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528442 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528445 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528448 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528451 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528454 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528456 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528459 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:39:44.641292 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528462 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528473 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528476 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528478 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528481 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528484 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528488 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528491 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528494 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528496 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528498 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528501 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528505 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528508 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528511 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528513 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528516 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528519 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528522 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:39:44.642162 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528524 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528527 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528530 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528532 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528535 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528538 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528540 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528543 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528545 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528548 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528550 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528553 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528555 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528558 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528561 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528564 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528566 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528569 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528573 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528577 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:39:44.643542 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528580 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528584 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528587 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528589 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528593 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528596 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528598 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528600 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528603 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528606 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528608 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528611 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528613 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528616 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528618 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528620 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528623 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528625 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528628 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528631 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:39:44.644460 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528633 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528636 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.528639 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529107 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529113 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529117 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529121 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529125 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529128 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529131 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529134 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529137 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529140 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529142 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529145 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529147 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529150 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529153 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529155 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:39:44.645815 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529158 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529160 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529163 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529166 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529168 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529171 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529173 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529176 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529178 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529181 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529184 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529186 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529189 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529192 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529195 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529197 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529200 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529202 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529205 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:39:44.646710 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529208 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529210 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529213 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529216 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529218 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529221 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529223 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529225 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529228 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529230 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529232 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529235 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529238 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529240 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529243 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529247 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529249 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529252 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529254 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529257 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:39:44.648519 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529260 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529262 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529265 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529267 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529270 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529273 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529275 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529278 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529280 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529284 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529288 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529290 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529293 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529296 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529299 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529302 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529304 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529307 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529312 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529315 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:39:44.649425 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529317 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529320 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529322 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529325 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529328 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529331 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529336 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529339 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529342 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529345 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529347 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529350 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529352 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529355 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529358 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529360 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529363 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529366 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529368 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:39:44.652121 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529372 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529375 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529378 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529380 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.529383 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529458 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529471 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529478 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529483 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529494 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529498 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529504 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529509 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529512 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529515 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529519 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529522 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529525 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529528 2562 flags.go:64] FLAG: --cgroup-root="" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529531 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529534 2562 flags.go:64] FLAG: --client-ca-file="" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529537 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529540 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529548 2562 flags.go:64] FLAG: --cluster-domain="" Jul 06 13:39:44.653254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529551 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529554 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529558 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529561 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529565 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529568 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529572 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529575 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529578 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529581 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529584 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529587 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529590 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529594 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529597 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529600 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529603 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529606 2562 flags.go:64] FLAG: --enable-server="true" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529609 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529618 2562 flags.go:64] FLAG: --event-burst="100" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529621 2562 flags.go:64] FLAG: --event-qps="50" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529624 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529627 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529630 2562 flags.go:64] FLAG: --eviction-hard="" Jul 06 13:39:44.654994 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529634 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529637 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529640 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529643 2562 flags.go:64] FLAG: --eviction-soft="" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529647 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529649 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529652 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529655 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529658 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529661 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529664 2562 flags.go:64] FLAG: --feature-gates="" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529670 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529674 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529677 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529680 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529683 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529686 2562 flags.go:64] FLAG: --help="false" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529689 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-134-202.ec2.internal" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529692 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529694 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529697 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529700 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529704 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529707 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 13:39:44.655871 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529710 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529713 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529716 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529721 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529725 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529728 2562 flags.go:64] FLAG: --kube-reserved="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529731 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529733 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529736 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529739 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529742 2562 flags.go:64] FLAG: --lock-file="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529745 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529748 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529751 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529756 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529759 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529762 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529764 2562 flags.go:64] FLAG: --logging-format="text" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529767 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529770 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529775 2562 flags.go:64] FLAG: --manifest-url="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529778 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529796 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529801 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529805 2562 flags.go:64] FLAG: --max-pods="110" Jul 06 13:39:44.657259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529809 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529812 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529815 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529818 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529820 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529824 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529827 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529834 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529838 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529841 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529846 2562 flags.go:64] FLAG: --pod-cidr="" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529849 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529854 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529857 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529860 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529863 2562 flags.go:64] FLAG: --port="10250" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529866 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529869 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-050e5f3b92e48608d" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529872 2562 flags.go:64] FLAG: --qos-reserved="" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529875 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529878 2562 flags.go:64] FLAG: --register-node="true" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529881 2562 flags.go:64] FLAG: --register-with-taints="" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529885 2562 flags.go:64] FLAG: --registry-burst="10" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529888 2562 flags.go:64] FLAG: --registry-qps="5" Jul 06 13:39:44.658597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529890 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529893 2562 flags.go:64] FLAG: --reserved-memory="" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529897 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529903 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529906 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529909 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529912 2562 flags.go:64] FLAG: --runonce="false" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529915 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529918 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529921 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529924 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529926 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529930 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529933 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529936 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529939 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529942 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529945 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529948 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529952 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529956 2562 flags.go:64] FLAG: --system-cgroups="" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529959 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529964 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529967 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529970 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 13:39:44.661775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529975 2562 flags.go:64] FLAG: --tls-min-version="" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529977 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529980 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529983 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529986 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529989 2562 flags.go:64] FLAG: --v="2" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529993 2562 flags.go:64] FLAG: --version="false" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.529997 2562 flags.go:64] FLAG: --vmodule="" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.530001 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.530004 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530132 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530136 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530139 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530142 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530144 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530147 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530149 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530152 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530155 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530157 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530160 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530163 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:39:44.663545 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530165 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530168 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530170 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530173 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530176 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530179 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530182 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530186 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530188 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530191 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530193 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530197 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530199 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530202 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530204 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530207 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530210 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530212 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530215 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530217 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:39:44.665340 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530221 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530224 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530227 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530230 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530233 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530237 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530240 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530243 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530245 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530248 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530251 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530253 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530255 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530258 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530261 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530263 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530267 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530269 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530272 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:39:44.666246 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530275 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530277 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530280 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530282 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530285 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530287 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530290 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530292 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530295 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530297 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530300 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530302 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530306 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530311 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530314 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530318 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530322 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530325 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530328 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:39:44.787274 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530331 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530333 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530336 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530339 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530341 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530344 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530346 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530349 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530351 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530354 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530358 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530362 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530364 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530367 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530370 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530372 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530375 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530378 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530380 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530383 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:39:44.788141 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530385 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530388 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530390 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530393 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530395 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530398 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530401 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530404 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.530406 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.531182 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.537948 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.537964 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.537985 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.537993 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538049 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:39:44.790402 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538054 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538057 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538060 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538063 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538066 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538069 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538071 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538074 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538077 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538080 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538083 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538086 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538090 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538093 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538095 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538098 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538101 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538104 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538106 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:39:44.791276 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538109 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538111 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538114 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538116 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538119 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538122 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538124 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538128 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538133 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538136 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538139 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538141 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538144 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538150 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538152 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538155 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538158 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538160 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538163 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538166 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:39:44.793925 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538168 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538171 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538175 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538178 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538180 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538183 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538186 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538191 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538195 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538198 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538200 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538203 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538206 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538209 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538211 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538214 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538216 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538219 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538222 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:39:44.794801 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538224 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538227 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538231 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538234 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538236 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538239 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538243 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538245 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538248 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538250 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538253 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538256 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538258 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538261 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538263 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538266 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538268 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538271 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538274 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538277 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538280 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:39:44.797467 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538283 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538285 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538288 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538290 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538293 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538296 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538298 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538301 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538303 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538306 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538308 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538311 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538313 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538317 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538320 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538323 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538325 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538330 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:39:44.798358 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538334 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.538339 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538453 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538458 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538461 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538464 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538467 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538470 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538472 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538475 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538478 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538480 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538483 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538486 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538489 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 13:39:44.801911 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538491 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538494 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538496 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538499 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538501 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538504 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538507 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538523 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538526 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538529 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538531 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538534 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538537 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538540 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538543 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538546 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538550 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538553 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538555 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 13:39:44.802782 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538558 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538561 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538563 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538566 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538569 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538571 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538575 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538579 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538581 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538584 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538587 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538590 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538594 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538596 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538599 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538601 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538604 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538606 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538609 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538612 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 13:39:44.805901 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538614 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538617 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538619 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538622 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538624 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538627 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538630 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538633 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538635 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538639 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538642 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538644 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538647 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538651 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538654 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538656 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538659 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538661 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538664 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538666 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 13:39:44.806762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538669 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538672 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538674 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538677 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538679 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538682 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538685 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538687 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538689 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538692 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538694 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538697 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538700 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538702 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538705 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538707 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538710 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538713 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538717 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538719 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 13:39:44.811659 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538722 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538727 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538731 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538734 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538737 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538740 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:44.538743 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.538748 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.538875 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.540993 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.542375 2562 server.go:1019] "Starting client certificate rotation" Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.542473 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.542522 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 13:39:44.812546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.544679 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.547734 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.560047 2562 log.go:25] "Validated CRI v1 runtime API" Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.562101 2562 log.go:25] "Validated CRI v1 image API" Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.565118 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.567397 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9001d090-f25a-454b-a126-809290cc6db0:/dev/nvme0n1p4 e8f7b443-bf94-4776-b483-303db289d639:/dev/nvme0n1p3] Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.567411 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 13:39:44.817443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.572344 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.572329 2562 manager.go:216] Machine: {Timestamp:2026-07-06 13:39:44.571045034 +0000 UTC m=+0.320858610 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101044 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec272937f123b98e3d26f0424b9eca85 SystemUUID:ec272937-f123-b98e-3d26-f0424b9eca85 BootID:a06ebacd-2b77-4984-9006-e4501883bb39 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e4:49:56:4f:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e4:49:56:4f:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:18:17:b4:d7:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.572468 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.572549 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.575534 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.575557 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.576251 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.576261 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.576269 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.576863 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.578164 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.595608 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79f5x" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.605069 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79f5x" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872113 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872280 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872315 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872327 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 13:39:44.934931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872340 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 13:39:44.884219 ip-10-0-134-202 systemd[1]: Started Kubernetes Kubelet. Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872352 2562 kubelet.go:398] "Adding apiserver pod source" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872373 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872832 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.872849 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.875122 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.877182 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.877219 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878480 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878496 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878502 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878509 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878523 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878533 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878542 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878548 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878564 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878570 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878579 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.878585 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.879422 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.879429 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.883178 2562 server.go:1268] "Started kubelet" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.883281 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.883397 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.883532 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.889048 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.890182 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:44.893244 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.893965 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.894160 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.894962 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.899544 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.899604 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:44.899743 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900473 2562 factory.go:55] Registering systemd factory Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900464 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900489 2562 factory.go:223] Registration of the systemd container factory successfully Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900506 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900519 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900661 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900668 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900737 2562 factory.go:153] Registering CRI-O factory Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900749 2562 factory.go:223] Registration of the crio container factory successfully Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900797 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900836 2562 factory.go:103] Registering Raw factory Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.900849 2562 manager.go:1196] Started watching for new ooms in manager Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.901298 2562 manager.go:349] Starting recovery of all containers Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.901979 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:44.905515 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-202.ec2.internal\" not found" node="ip-10-0-134-202.ec2.internal" Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.912401 2562 manager.go:354] Recovery completed Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.916324 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:44.936161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.917227 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919246 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919277 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919291 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919853 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919867 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919865 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.919894 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.921126 2562 policy_none.go:49] "None policy: Start" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.921140 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.921151 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 13:39:44.941160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.922101 2562 policy_none.go:47] "Start" Jul 06 13:39:45.083173 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:44.973998 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:45.083173 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.000158 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:45.209898 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.100799 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:45.209898 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.201243 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.252979 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.253041 2562 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.291920 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.302139 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.303159 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.303186 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.303199 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:39:45.329418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.309290 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:45.375720 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.375684 2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-202.ec2.internal" not found Jul 06 13:39:45.396701 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.396518 2562 manager.go:337] "Starting Device Plugin manager" Jul 06 13:39:45.396853 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.396714 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 13:39:45.396853 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.396730 2562 server.go:87] "Starting device plugin registration server" Jul 06 13:39:45.397028 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.397014 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 13:39:45.397084 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.397030 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 13:39:45.397178 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.397152 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 13:39:45.397275 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.397262 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 13:39:45.397340 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.397276 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 13:39:45.398319 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.398300 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 13:39:45.398418 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.398345 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:45.498294 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.498232 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 13:39:45.500505 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.499505 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 13:39:45.500505 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.499542 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 13:39:45.500505 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.499555 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 13:39:45.500505 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.499582 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.511193 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.511170 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.511300 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.511199 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-202.ec2.internal\": node \"ip-10-0-134-202.ec2.internal\" not found" Jul 06 13:39:45.535636 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.535610 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 13:39:45.537056 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.537038 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 13:39:45.537161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.537086 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 13:39:45.537161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.537113 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 13:39:45.537161 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.537148 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 13:39:45.539438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.539421 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:45.541798 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.541782 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 13:39:45.541917 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.541898 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 13:39:45.541988 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.541913 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 13:39:45.541988 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.541913 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 13:39:45.637275 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.637223 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal"] Jul 06 13:39:45.640728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.640711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.642789 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.642773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.660498 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.660474 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.664670 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.664655 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.673565 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.673550 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 13:39:45.673741 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.673726 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 13:39:45.703988 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.703956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.704076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.703995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.704127 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.704052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b12aea9bf8b99530f8679e8c6d1d16c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"4b12aea9bf8b99530f8679e8c6d1d16c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.804244 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.804172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.804244 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.804204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b12aea9bf8b99530f8679e8c6d1d16c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"4b12aea9bf8b99530f8679e8c6d1d16c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.804244 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.804225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.804438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.804271 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.804438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.804290 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ee03c06c928493e4823bc059365a1f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal\" (UID: \"3ee03c06c928493e4823bc059365a1f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.804438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.804314 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b12aea9bf8b99530f8679e8c6d1d16c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-202.ec2.internal\" (UID: \"4b12aea9bf8b99530f8679e8c6d1d16c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.875315 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.875282 2562 apiserver.go:52] "Watching apiserver" Jul 06 13:39:45.875315 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.875263 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 13:34:45 +0000 UTC" deadline="2028-02-15 00:08:34.814959383 +0000 UTC" Jul 06 13:39:45.875315 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.875315 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14122h28m48.939647945s" Jul 06 13:39:45.882663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.882633 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 13:39:45.883053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.883031 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-p27hl","openshift-multus/network-metrics-daemon-ntd56","openshift-ovn-kubernetes/ovnkube-node-6vsnb","kube-system/konnectivity-agent-5g528","kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9tdgh","openshift-dns/node-resolver-zpc42","openshift-image-registry/node-ca-5ptzd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal","openshift-network-diagnostics/network-check-target-jsj4h","openshift-network-operator/iptables-alerter-8kcd9","kube-system/global-pull-secret-syncer-8nr9n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8","openshift-multus/multus-6t775"] Jul 06 13:39:45.885791 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.885775 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.887960 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.887938 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:45.888079 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.888012 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:45.889073 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.889055 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z7thk\"" Jul 06 13:39:45.889176 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.889063 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 13:39:45.889338 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.889322 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 13:39:45.889409 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.889382 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.889467 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.889409 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 13:39:45.889520 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.889503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.890361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.890344 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.893203 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.893181 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:45.894358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894186 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 13:39:45.894358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894242 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 13:39:45.894358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894252 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 13:39:45.894358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.894709 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j49zs\"" Jul 06 13:39:45.894709 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894582 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 13:39:45.894709 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.894623 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.895395 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.895378 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.895627 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.895606 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 13:39:45.895996 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.895981 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sc6fj\"" Jul 06 13:39:45.896062 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.896017 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 13:39:45.897981 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.897939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:45.898267 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.898058 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.898267 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.897941 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wgf6c\"" Jul 06 13:39:45.898267 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.898082 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.900287 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.900268 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 13:39:45.900383 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.900323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:45.900511 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.900495 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.900684 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.900665 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xbv25\"" Jul 06 13:39:45.900781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.900766 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.902582 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.902567 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:45.902649 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.902624 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:45.903048 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.903029 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.903111 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.903032 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.903111 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.903061 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 13:39:45.903111 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.903056 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hksb5\"" Jul 06 13:39:45.904340 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovn-node-metrics-cert\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.904414 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904348 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-lib-modules\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.904414 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlzx\" (UniqueName: \"kubernetes.io/projected/4b0035b3-2906-47b0-924f-7ffd405ea17d-kube-api-access-xtlzx\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.904414 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/883b955a-f5b4-4cae-87e5-55ea75c2be59-hosts-file\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:45.904530 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883b955a-f5b4-4cae-87e5-55ea75c2be59-tmp-dir\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:45.904530 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.904530 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00bf2769-4a95-4dc6-b1c5-825808185157-agent-certs\") pod \"konnectivity-agent-5g528\" (UID: \"00bf2769-4a95-4dc6-b1c5-825808185157\") " pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:45.904661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00bf2769-4a95-4dc6-b1c5-825808185157-konnectivity-ca\") pod \"konnectivity-agent-5g528\" (UID: \"00bf2769-4a95-4dc6-b1c5-825808185157\") " pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:45.904661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe5c8515-7f08-4442-ba21-c438df4a8849-serviceca\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:45.904661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7vh\" (UniqueName: \"kubernetes.io/projected/fd5b0edf-c431-4ee8-b16d-27147bc509a4-kube-api-access-qr7vh\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:45.904661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-run-netns\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.904661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904636 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-log-socket\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdv4r\" (UniqueName: \"kubernetes.io/projected/883b955a-f5b4-4cae-87e5-55ea75c2be59-kube-api-access-pdv4r\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-os-release\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5qx\" (UniqueName: \"kubernetes.io/projected/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-kube-api-access-9m5qx\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe5c8515-7f08-4442-ba21-c438df4a8849-host\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hds\" (UniqueName: \"kubernetes.io/projected/fe5c8515-7f08-4442-ba21-c438df4a8849-kube-api-access-m7hds\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-kubelet\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904866 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:45.904916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cnibin\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.904982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-slash\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysctl-d\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-systemd\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-run\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-sys\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-var-lib-kubelet\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-etc-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-ovn\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysctl-conf\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-tuned\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cni-binary-copy\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.905305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-systemd\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-cni-netd\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxfw\" (UniqueName: \"kubernetes.io/projected/b170b40f-55ae-4016-a4be-f8ca0723f02c-kube-api-access-pnxfw\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-modprobe-d\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysconfig\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-kubernetes\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-host\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-env-overrides\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b0035b3-2906-47b0-924f-7ffd405ea17d-tmp\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905515 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-system-cni-dir\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-var-lib-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-node-log\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-cni-bin\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovnkube-config\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovnkube-script-lib\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.905886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.905659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-systemd-units\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:45.907302 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.907285 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.907302 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.907287 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:45.907449 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:45.907351 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:45.907449 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.907359 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 13:39:45.907608 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.907592 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v2g9f\"" Jul 06 13:39:45.907698 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.907684 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.909882 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.909860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:45.912364 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.912342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6t775" Jul 06 13:39:45.912483 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.912461 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 13:39:45.912576 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.912555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 13:39:45.913515 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.913499 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 13:39:45.913619 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.913504 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bc2ps\"" Jul 06 13:39:45.915148 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.915130 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 13:39:45.915685 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.915668 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7zd4x\"" Jul 06 13:39:45.918254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.918233 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 13:39:45.939135 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.939111 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj65n" Jul 06 13:39:45.952066 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.952048 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj65n" Jul 06 13:39:45.978054 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.978018 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" Jul 06 13:39:45.978158 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:45.978071 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" Jul 06 13:39:46.001489 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.001463 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 13:39:46.006418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006392 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-node-log\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.006548 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovnkube-config\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.006548 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-lib-modules\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.006548 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006516 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-node-log\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.006712 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006571 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-lib-modules\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.006712 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6l27\" (UniqueName: \"kubernetes.io/projected/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-kube-api-access-f6l27\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.006712 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-multus-certs\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.006712 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxxs\" (UniqueName: \"kubernetes.io/projected/fa434433-f3d7-457d-a191-0e421b7de99f-kube-api-access-jjxxs\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.006938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.006938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7vh\" (UniqueName: \"kubernetes.io/projected/fd5b0edf-c431-4ee8-b16d-27147bc509a4-kube-api-access-qr7vh\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:46.006938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdv4r\" (UniqueName: \"kubernetes.io/projected/883b955a-f5b4-4cae-87e5-55ea75c2be59-kube-api-access-pdv4r\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.006938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-socket-dir-parent\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.006938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-cni-multus\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlzx\" (UniqueName: \"kubernetes.io/projected/4b0035b3-2906-47b0-924f-7ffd405ea17d-kube-api-access-xtlzx\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.006969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-os-release\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5qx\" (UniqueName: \"kubernetes.io/projected/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-kube-api-access-9m5qx\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007016 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe5c8515-7f08-4442-ba21-c438df4a8849-host\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hds\" (UniqueName: \"kubernetes.io/projected/fe5c8515-7f08-4442-ba21-c438df4a8849-kube-api-access-m7hds\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysctl-d\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-systemd\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.007197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007177 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovnkube-config\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.007549 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-sys\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.007549 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3114dd-57c2-4a17-ac11-334b98b10191-dbus\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.007549 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007527 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe5c8515-7f08-4442-ba21-c438df4a8849-host\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.007695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-registration-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.007695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-os-release\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.007695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007636 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-os-release\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.007695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007638 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-systemd\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.007695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007652 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysctl-d\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.007695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-netns\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007710 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007707 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-sys\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-etc-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-ovn\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-etc-kubernetes\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.007990 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7smrx\" (UniqueName: \"kubernetes.io/projected/91a38853-d604-4c71-bac6-edfe16a39b66-kube-api-access-7smrx\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b0035b3-2906-47b0-924f-7ffd405ea17d-tmp\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008015 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-etc-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008068 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-ovn\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-systemd\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008115 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-systemd\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-cni-netd\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-kubernetes\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008244 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-cni-netd\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-host\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008304 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-kubernetes\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-device-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-hostroot\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008382 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-host\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008407 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-env-overrides\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-sys-fs\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-cni-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008494 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91a38853-d604-4c71-bac6-edfe16a39b66-multus-daemon-config\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-system-cni-dir\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-var-lib-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.008616 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-cni-bin\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovnkube-script-lib\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008707 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-system-cni-dir\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008675 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-var-lib-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008729 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883b955a-f5b4-4cae-87e5-55ea75c2be59-tmp-dir\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008766 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-cni-bin\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-k8s-cni-cncf-io\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008844 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-run-openvswitch\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-systemd-units\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008925 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-env-overrides\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovn-node-metrics-cert\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.008981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-system-cni-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009002 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-systemd-units\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009009 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-cni-bin\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-kubelet\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00bf2769-4a95-4dc6-b1c5-825808185157-agent-certs\") pod \"konnectivity-agent-5g528\" (UID: \"00bf2769-4a95-4dc6-b1c5-825808185157\") " pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:46.009596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00bf2769-4a95-4dc6-b1c5-825808185157-konnectivity-ca\") pod \"konnectivity-agent-5g528\" (UID: \"00bf2769-4a95-4dc6-b1c5-825808185157\") " pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009120 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/883b955a-f5b4-4cae-87e5-55ea75c2be59-tmp-dir\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe5c8515-7f08-4442-ba21-c438df4a8849-serviceca\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-run-netns\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-log-socket\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/883b955a-f5b4-4cae-87e5-55ea75c2be59-hosts-file\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-kubelet\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009327 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovnkube-script-lib\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009401 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-log-socket\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009422 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/883b955a-f5b4-4cae-87e5-55ea75c2be59-hosts-file\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009483 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-run-netns\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-run\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-var-lib-kubelet\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-host-slash\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cnibin\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.010852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-slash\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009706 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-kubelet\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009721 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00bf2769-4a95-4dc6-b1c5-825808185157-konnectivity-ca\") pod \"konnectivity-agent-5g528\" (UID: \"00bf2769-4a95-4dc6-b1c5-825808185157\") " pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-tuned\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91a38853-d604-4c71-bac6-edfe16a39b66-cni-binary-copy\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysctl-conf\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3114dd-57c2-4a17-ac11-334b98b10191-kubelet-config\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-cnibin\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cni-binary-copy\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxfw\" (UniqueName: \"kubernetes.io/projected/b170b40f-55ae-4016-a4be-f8ca0723f02c-kube-api-access-pnxfw\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010104 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-run\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010113 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysctl-conf\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-modprobe-d\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysconfig\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.011686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010171 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe5c8515-7f08-4442-ba21-c438df4a8849-serviceca\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-socket-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.010226 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010274 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-var-lib-kubelet\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-iptables-alerter-script\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.009961 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.010359 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:46.510294757 +0000 UTC m=+2.260108338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010364 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-modprobe-d\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010424 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-sysconfig\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010535 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b170b40f-55ae-4016-a4be-f8ca0723f02c-host-slash\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010741 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cni-binary-copy\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010822 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-cnibin\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010825 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.010985 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.011022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.012538 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.011049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-conf-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.013492 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.013293 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b170b40f-55ae-4016-a4be-f8ca0723f02c-ovn-node-metrics-cert\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.013843 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.013823 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00bf2769-4a95-4dc6-b1c5-825808185157-agent-certs\") pod \"konnectivity-agent-5g528\" (UID: \"00bf2769-4a95-4dc6-b1c5-825808185157\") " pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:46.014851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.014828 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b0035b3-2906-47b0-924f-7ffd405ea17d-tmp\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.015039 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.015013 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b0035b3-2906-47b0-924f-7ffd405ea17d-etc-tuned\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.016080 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.016057 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:39:46.016178 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.016087 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:39:46.016178 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.016103 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:46.016290 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.016201 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:39:46.516180593 +0000 UTC m=+2.265994139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:46.016816 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.016775 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7vh\" (UniqueName: \"kubernetes.io/projected/fd5b0edf-c431-4ee8-b16d-27147bc509a4-kube-api-access-qr7vh\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:46.017146 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.017122 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5qx\" (UniqueName: \"kubernetes.io/projected/23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300-kube-api-access-9m5qx\") pod \"multus-additional-cni-plugins-p27hl\" (UID: \"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300\") " pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.017253 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.017171 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdv4r\" (UniqueName: \"kubernetes.io/projected/883b955a-f5b4-4cae-87e5-55ea75c2be59-kube-api-access-pdv4r\") pod \"node-resolver-zpc42\" (UID: \"883b955a-f5b4-4cae-87e5-55ea75c2be59\") " pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.017959 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.017824 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hds\" (UniqueName: \"kubernetes.io/projected/fe5c8515-7f08-4442-ba21-c438df4a8849-kube-api-access-m7hds\") pod \"node-ca-5ptzd\" (UID: \"fe5c8515-7f08-4442-ba21-c438df4a8849\") " pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.018672 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.018650 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxfw\" (UniqueName: \"kubernetes.io/projected/b170b40f-55ae-4016-a4be-f8ca0723f02c-kube-api-access-pnxfw\") pod \"ovnkube-node-6vsnb\" (UID: \"b170b40f-55ae-4016-a4be-f8ca0723f02c\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.019061 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.019032 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlzx\" (UniqueName: \"kubernetes.io/projected/4b0035b3-2906-47b0-924f-7ffd405ea17d-kube-api-access-xtlzx\") pod \"tuned-9tdgh\" (UID: \"4b0035b3-2906-47b0-924f-7ffd405ea17d\") " pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.112339 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-etc-kubernetes\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112339 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7smrx\" (UniqueName: \"kubernetes.io/projected/91a38853-d604-4c71-bac6-edfe16a39b66-kube-api-access-7smrx\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112339 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-device-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.112339 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112316 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-hostroot\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112339 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-sys-fs\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112369 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-etc-kubernetes\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-cni-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112408 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-hostroot\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91a38853-d604-4c71-bac6-edfe16a39b66-multus-daemon-config\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112452 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-sys-fs\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-k8s-cni-cncf-io\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112506 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-k8s-cni-cncf-io\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-system-cni-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-cni-bin\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112541 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-device-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-kubelet\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112541 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-cni-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112590 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-cni-bin\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112596 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-system-cni-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-host-slash\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112632 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-kubelet\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.112683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112647 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-host-slash\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91a38853-d604-4c71-bac6-edfe16a39b66-cni-binary-copy\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112731 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3114dd-57c2-4a17-ac11-334b98b10191-kubelet-config\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-cnibin\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112786 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a3114dd-57c2-4a17-ac11-334b98b10191-kubelet-config\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-socket-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-iptables-alerter-script\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112858 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-cnibin\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112885 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-conf-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112977 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-socket-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.112993 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-conf-dir\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113042 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.113096 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6l27\" (UniqueName: \"kubernetes.io/projected/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-kube-api-access-f6l27\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-multus-certs\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.113585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxxs\" (UniqueName: \"kubernetes.io/projected/fa434433-f3d7-457d-a191-0e421b7de99f-kube-api-access-jjxxs\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113096 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91a38853-d604-4c71-bac6-edfe16a39b66-multus-daemon-config\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.113164 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:46.613145432 +0000 UTC m=+2.362958977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113204 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-multus-certs\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-socket-dir-parent\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-cni-multus\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113260 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-multus-socket-dir-parent\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3114dd-57c2-4a17-ac11-334b98b10191-dbus\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-registration-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113325 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-os-release\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113296 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-var-lib-cni-multus\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-netns\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113354 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a3114dd-57c2-4a17-ac11-334b98b10191-dbus\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113341 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91a38853-d604-4c71-bac6-edfe16a39b66-cni-binary-copy\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113414 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa434433-f3d7-457d-a191-0e421b7de99f-registration-dir\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113418 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-host-run-netns\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113441 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/91a38853-d604-4c71-bac6-edfe16a39b66-os-release\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.114138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.113479 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-iptables-alerter-script\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.124835 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.124802 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6l27\" (UniqueName: \"kubernetes.io/projected/8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803-kube-api-access-f6l27\") pod \"iptables-alerter-8kcd9\" (UID: \"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803\") " pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.124986 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.124908 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxxs\" (UniqueName: \"kubernetes.io/projected/fa434433-f3d7-457d-a191-0e421b7de99f-kube-api-access-jjxxs\") pod \"aws-ebs-csi-driver-node-4qtv8\" (UID: \"fa434433-f3d7-457d-a191-0e421b7de99f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.124986 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.124922 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7smrx\" (UniqueName: \"kubernetes.io/projected/91a38853-d604-4c71-bac6-edfe16a39b66-kube-api-access-7smrx\") pod \"multus-6t775\" (UID: \"91a38853-d604-4c71-bac6-edfe16a39b66\") " pod="openshift-multus/multus-6t775" Jul 06 13:39:46.212089 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.212051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p27hl" Jul 06 13:39:46.219828 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.219792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:39:46.248023 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.247998 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5g528" Jul 06 13:39:46.266314 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.266290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" Jul 06 13:39:46.281014 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.280988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpc42" Jul 06 13:39:46.292576 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.292552 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ptzd" Jul 06 13:39:46.308108 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.308086 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kcd9" Jul 06 13:39:46.338720 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.338697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" Jul 06 13:39:46.344539 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.344515 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6t775" Jul 06 13:39:46.517002 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.516917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:46.517255 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.517016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:46.517255 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.517143 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:46.517255 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.517213 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:47.51719516 +0000 UTC m=+3.267008709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:46.517255 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.517146 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:39:46.517255 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.517257 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:39:46.517473 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.517272 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:46.517473 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.517333 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:39:47.517315303 +0000 UTC m=+3.267128863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:46.552604 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.552575 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:46.593255 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.593215 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee03c06c928493e4823bc059365a1f8.slice/crio-d8b782ebe10f0130b8dc5227a34330ab76ed95cf7bdc58a5d5be17bda5f63df8 WatchSource:0}: Error finding container d8b782ebe10f0130b8dc5227a34330ab76ed95cf7bdc58a5d5be17bda5f63df8: Status 404 returned error can't find the container with id d8b782ebe10f0130b8dc5227a34330ab76ed95cf7bdc58a5d5be17bda5f63df8 Jul 06 13:39:46.593534 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.593510 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b12aea9bf8b99530f8679e8c6d1d16c.slice/crio-6edfaaa92ef55bd39d1c209dccf75a456c8a8054edc473f390ad7785f14c5f43 WatchSource:0}: Error finding container 6edfaaa92ef55bd39d1c209dccf75a456c8a8054edc473f390ad7785f14c5f43: Status 404 returned error can't find the container with id 6edfaaa92ef55bd39d1c209dccf75a456c8a8054edc473f390ad7785f14c5f43 Jul 06 13:39:46.598939 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.598915 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:39:46.617915 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.617874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:46.618074 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.618019 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:46.618131 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:46.618090 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:47.618070706 +0000 UTC m=+3.367884252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:46.683300 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.683273 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf2769_4a95_4dc6_b1c5_825808185157.slice/crio-1f746cd5a8c62dadde2ed619a54d5fedb05acab5e4ffd245bb65e4a247ef1814 WatchSource:0}: Error finding container 1f746cd5a8c62dadde2ed619a54d5fedb05acab5e4ffd245bb65e4a247ef1814: Status 404 returned error can't find the container with id 1f746cd5a8c62dadde2ed619a54d5fedb05acab5e4ffd245bb65e4a247ef1814 Jul 06 13:39:46.685320 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.685299 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5c8515_7f08_4442_ba21_c438df4a8849.slice/crio-f695c9310e87fdd70988a9353693ba482189b01f4506010b14ae2aca03b660da WatchSource:0}: Error finding container f695c9310e87fdd70988a9353693ba482189b01f4506010b14ae2aca03b660da: Status 404 returned error can't find the container with id f695c9310e87fdd70988a9353693ba482189b01f4506010b14ae2aca03b660da Jul 06 13:39:46.695838 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.695812 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f554ffa_d2c3_4cdf_8f37_2e4a8ac00803.slice/crio-1cc0ec0adf92e6ffd3b8f064e05dabb170d3cc09c30ef94b30425ceddda23d74 WatchSource:0}: Error finding container 1cc0ec0adf92e6ffd3b8f064e05dabb170d3cc09c30ef94b30425ceddda23d74: Status 404 returned error can't find the container with id 1cc0ec0adf92e6ffd3b8f064e05dabb170d3cc09c30ef94b30425ceddda23d74 Jul 06 13:39:46.722363 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.722335 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb170b40f_55ae_4016_a4be_f8ca0723f02c.slice/crio-5fdb3fb5db1e751affa64a35023b719ae34c4cbcd08257262e631920367e8483 WatchSource:0}: Error finding container 5fdb3fb5db1e751affa64a35023b719ae34c4cbcd08257262e631920367e8483: Status 404 returned error can't find the container with id 5fdb3fb5db1e751affa64a35023b719ae34c4cbcd08257262e631920367e8483 Jul 06 13:39:46.723006 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.722980 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883b955a_f5b4_4cae_87e5_55ea75c2be59.slice/crio-7dcb6b10b58ef5a6e2b30ff002c921a1c1633bcc2e006d853132229f6f570589 WatchSource:0}: Error finding container 7dcb6b10b58ef5a6e2b30ff002c921a1c1633bcc2e006d853132229f6f570589: Status 404 returned error can't find the container with id 7dcb6b10b58ef5a6e2b30ff002c921a1c1633bcc2e006d853132229f6f570589 Jul 06 13:39:46.742176 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.742155 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa434433_f3d7_457d_a191_0e421b7de99f.slice/crio-c5480c6990603d14f0aa2b092a939133cefb25b1e31385e09aa560ffbf01839a WatchSource:0}: Error finding container c5480c6990603d14f0aa2b092a939133cefb25b1e31385e09aa560ffbf01839a: Status 404 returned error can't find the container with id c5480c6990603d14f0aa2b092a939133cefb25b1e31385e09aa560ffbf01839a Jul 06 13:39:46.768430 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.768375 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dcdf1e_4fb7_46ef_9ffc_8d5d0ddc2300.slice/crio-221285f784d44c0d940f21073a3f9cdc2c70d566ef92b55ee0050a5d6c87cf74 WatchSource:0}: Error finding container 221285f784d44c0d940f21073a3f9cdc2c70d566ef92b55ee0050a5d6c87cf74: Status 404 returned error can't find the container with id 221285f784d44c0d940f21073a3f9cdc2c70d566ef92b55ee0050a5d6c87cf74 Jul 06 13:39:46.815758 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.815722 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:46.879415 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:39:46.879378 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0035b3_2906_47b0_924f_7ffd405ea17d.slice/crio-a300b2cad2ac63b7f105b40831804c9a8c750499f13014d5f3a19e96495df03a WatchSource:0}: Error finding container a300b2cad2ac63b7f105b40831804c9a8c750499f13014d5f3a19e96495df03a: Status 404 returned error can't find the container with id a300b2cad2ac63b7f105b40831804c9a8c750499f13014d5f3a19e96495df03a Jul 06 13:39:46.955725 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.955663 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 13:34:46 +0000 UTC" deadline="2028-02-01 19:32:59.129620414 +0000 UTC" Jul 06 13:39:46.955725 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:46.955716 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13805h53m12.173907491s" Jul 06 13:39:47.104881 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.104845 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.532128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.532192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.532325 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.532393 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:49.532372185 +0000 UTC m=+5.282185750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.532892 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.532916 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.532931 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:47.533076 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.532985 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:39:49.532965424 +0000 UTC m=+5.282778995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:47.539830 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.538974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:47.539830 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.539091 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:47.539830 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.539187 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:47.539830 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.539265 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:47.539830 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.539563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:47.539830 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.539664 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:47.562412 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.562318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerStarted","Data":"221285f784d44c0d940f21073a3f9cdc2c70d566ef92b55ee0050a5d6c87cf74"} Jul 06 13:39:47.578736 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.578602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"5fdb3fb5db1e751affa64a35023b719ae34c4cbcd08257262e631920367e8483"} Jul 06 13:39:47.598829 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.598749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ptzd" event={"ID":"fe5c8515-7f08-4442-ba21-c438df4a8849","Type":"ContainerStarted","Data":"f695c9310e87fdd70988a9353693ba482189b01f4506010b14ae2aca03b660da"} Jul 06 13:39:47.614899 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.614793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" event={"ID":"4b12aea9bf8b99530f8679e8c6d1d16c","Type":"ContainerStarted","Data":"6edfaaa92ef55bd39d1c209dccf75a456c8a8054edc473f390ad7785f14c5f43"} Jul 06 13:39:47.632371 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.632326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"3ee03c06c928493e4823bc059365a1f8","Type":"ContainerStarted","Data":"d8b782ebe10f0130b8dc5227a34330ab76ed95cf7bdc58a5d5be17bda5f63df8"} Jul 06 13:39:47.632767 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.632739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:47.632923 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.632902 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:47.632997 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:47.632985 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:49.632964473 +0000 UTC m=+5.382778025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:47.656753 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.656687 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6t775" event={"ID":"91a38853-d604-4c71-bac6-edfe16a39b66","Type":"ContainerStarted","Data":"c16d83a289686b3f96201cf83027c1b8d2ab7556f646850c5715780db45b9d79"} Jul 06 13:39:47.687047 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.687007 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" event={"ID":"4b0035b3-2906-47b0-924f-7ffd405ea17d","Type":"ContainerStarted","Data":"a300b2cad2ac63b7f105b40831804c9a8c750499f13014d5f3a19e96495df03a"} Jul 06 13:39:47.728199 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.728130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" event={"ID":"fa434433-f3d7-457d-a191-0e421b7de99f","Type":"ContainerStarted","Data":"c5480c6990603d14f0aa2b092a939133cefb25b1e31385e09aa560ffbf01839a"} Jul 06 13:39:47.781013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.780886 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpc42" event={"ID":"883b955a-f5b4-4cae-87e5-55ea75c2be59","Type":"ContainerStarted","Data":"7dcb6b10b58ef5a6e2b30ff002c921a1c1633bcc2e006d853132229f6f570589"} Jul 06 13:39:47.802198 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.802099 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kcd9" event={"ID":"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803","Type":"ContainerStarted","Data":"1cc0ec0adf92e6ffd3b8f064e05dabb170d3cc09c30ef94b30425ceddda23d74"} Jul 06 13:39:47.841185 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.841135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5g528" event={"ID":"00bf2769-4a95-4dc6-b1c5-825808185157","Type":"ContainerStarted","Data":"1f746cd5a8c62dadde2ed619a54d5fedb05acab5e4ffd245bb65e4a247ef1814"} Jul 06 13:39:47.957213 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.957048 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 13:34:46 +0000 UTC" deadline="2028-03-11 05:42:59.947378142 +0000 UTC" Jul 06 13:39:47.957213 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:47.957165 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14728h3m11.990218929s" Jul 06 13:39:49.540735 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:49.540700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:49.541476 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.540849 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:49.541476 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:49.541188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:49.541476 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.541298 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:49.541476 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:49.541359 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:49.541476 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.541431 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:49.552183 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:49.552151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:49.552370 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:49.552204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:49.552370 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.552347 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:49.552496 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.552392 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:39:49.552496 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.552409 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:53.552389341 +0000 UTC m=+9.302202906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:49.552496 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.552414 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:39:49.552496 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.552427 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:49.552496 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.552487 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:39:53.552470275 +0000 UTC m=+9.302283832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:49.653007 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:49.652968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:49.653210 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.653132 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:49.653210 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:49.653206 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:39:53.653176329 +0000 UTC m=+9.402989900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:51.538390 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:51.538327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:51.538939 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:51.538492 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:51.538939 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:51.538899 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:51.539073 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:51.539014 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:51.539136 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:51.539074 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:51.539196 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:51.539152 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:53.538704 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:53.538144 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:53.538704 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:53.538191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:53.538704 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.538300 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:53.538704 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:53.538388 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:53.538704 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.538580 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:53.538704 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.538696 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:53.588679 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:53.588641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:53.588883 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:53.588689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:53.588883 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.588854 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:53.588883 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.588868 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:39:53.589068 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.588892 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:39:53.589068 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.588907 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:53.589068 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.588919 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:01.588901805 +0000 UTC m=+17.338715374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:39:53.589068 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.588963 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:40:01.588946586 +0000 UTC m=+17.338760143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:39:53.690019 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:53.689981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:53.690212 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.690160 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:53.690280 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:53.690232 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:01.690209473 +0000 UTC m=+17.440023022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:39:55.538499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:55.538456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:55.539024 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:55.538594 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:55.539024 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:55.538865 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:55.539024 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:55.539000 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:55.539210 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:55.539039 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:55.539210 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:55.539175 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:57.538365 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:57.538272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:57.538365 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:57.538318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:57.538931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:57.538272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:57.538931 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:57.538396 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:57.538931 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:57.538499 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:39:57.538931 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:57.538599 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:59.538227 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:59.537991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:39:59.538751 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:59.537992 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:39:59.538751 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:59.538324 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:39:59.538751 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:39:59.538045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:39:59.538751 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:59.538455 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:39:59.538751 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:39:59.538533 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:01.538163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:01.538125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:01.538623 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:01.538169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:01.538623 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:01.538125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:01.538623 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.538256 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:01.538623 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.538335 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:01.538623 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.538453 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:01.654091 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:01.654055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:01.654276 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:01.654101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:01.654276 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.654216 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:40:01.654276 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.654240 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:40:01.654276 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.654266 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:40:01.654276 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.654280 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:40:01.654485 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.654286 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:17.654264164 +0000 UTC m=+33.404077728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:40:01.654485 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.654333 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:40:17.654318671 +0000 UTC m=+33.404132215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:40:01.755417 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:01.755327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:01.755604 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.755472 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:40:01.755604 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:01.755540 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:17.755519783 +0000 UTC m=+33.505333344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:40:03.537366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:03.537328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:03.537366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:03.537360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:03.537898 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:03.537331 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:03.537898 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:03.537447 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:03.537898 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:03.537540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:03.537898 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:03.537615 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:05.538487 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:05.538450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:05.539015 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:05.538569 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:05.539015 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:05.538954 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:05.539132 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:05.539078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:05.539132 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:05.539096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:05.539233 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:05.539145 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:07.537733 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.537711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:07.538158 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.537731 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:07.538158 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:07.537846 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:07.538158 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:07.537903 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:07.538158 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.537945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:07.538158 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:07.538079 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:07.889148 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.889113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6t775" event={"ID":"91a38853-d604-4c71-bac6-edfe16a39b66","Type":"ContainerStarted","Data":"cac2341e52ce3757f6ec0013d787da75a8f332799a3db3160cb6a3a25d87ce55"} Jul 06 13:40:07.890879 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.890850 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" event={"ID":"4b0035b3-2906-47b0-924f-7ffd405ea17d","Type":"ContainerStarted","Data":"bdb962eb3df02e1e38ee1a8d8bda59ee964c4462ec308a523aad1e9a31a061ef"} Jul 06 13:40:07.893636 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.893612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"73da0c163c6d55e00f480d23f6196046d7f0f16477f2077ac43cdb2da3a4fcdc"} Jul 06 13:40:07.893741 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.893694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"b807d7476f9a808e5ff41243896f6010a16538b46bd5997b4b7d2b3d2d6bcbe2"} Jul 06 13:40:07.901619 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.901583 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" event={"ID":"4b12aea9bf8b99530f8679e8c6d1d16c","Type":"ContainerStarted","Data":"a2d3f3f0a1158b3d3e914ee691c55e7f460b91f0e74fc7808f80cd678d7ba1ff"} Jul 06 13:40:07.908505 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.908463 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6t775" podStartSLOduration=1.441676909 podStartE2EDuration="21.908447429s" totalImagesPullingTime="20.46677052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:07.907700071 +0000 UTC m=+23.657513636" watchObservedRunningTime="2026-07-06 13:40:07.908447429 +0000 UTC m=+23.658260994" Jul 06 13:40:07.923366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.923225 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" podStartSLOduration=21.923205037 podStartE2EDuration="21.923205037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:07.92311092 +0000 UTC m=+23.672924554" watchObservedRunningTime="2026-07-06 13:40:07.923205037 +0000 UTC m=+23.673018603" Jul 06 13:40:07.943173 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:07.943117 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9tdgh" podStartSLOduration=1.3023407869999999 podStartE2EDuration="21.94310278s" totalImagesPullingTime="20.640761993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:07.942437486 +0000 UTC m=+23.692251051" watchObservedRunningTime="2026-07-06 13:40:07.94310278 +0000 UTC m=+23.692916344" Jul 06 13:40:08.905927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.905885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpc42" event={"ID":"883b955a-f5b4-4cae-87e5-55ea75c2be59","Type":"ContainerStarted","Data":"68835d8e279b785e0b01c438bd3a28c400cb28a705d3fe19ca2bc1129a7fcb53"} Jul 06 13:40:08.910694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.910461 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kcd9" event={"ID":"8f554ffa-d2c3-4cdf-8f37-2e4a8ac00803","Type":"ContainerStarted","Data":"ca9a505bea42145949bda4867e23cd0a6ccc87dd961d7c74b71747116687aab0"} Jul 06 13:40:08.912194 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.912170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5g528" event={"ID":"00bf2769-4a95-4dc6-b1c5-825808185157","Type":"ContainerStarted","Data":"a540858bcdf0edb145a457ad76322449658515317ff8746f8fbc2d58d4aa6eb5"} Jul 06 13:40:08.913749 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.913728 2562 generic.go:356] "Generic (PLEG): container finished" podID="23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300" containerID="ec67340451939e8261c02b84a4c8f33db48d7d70f954d109c2ec397d1c8b1cde" exitCode=0 Jul 06 13:40:08.913859 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.913819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerDied","Data":"ec67340451939e8261c02b84a4c8f33db48d7d70f954d109c2ec397d1c8b1cde"} Jul 06 13:40:08.917291 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.917260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"aef769ab1f76d5c81462b23b4e4727b0bd0936887b54d55f37ffe481074747e2"} Jul 06 13:40:08.917387 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.917335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"0a61447d2631a80be2446eb98b050855ee22368ac0e7ae2cc0f0a60ff579a997"} Jul 06 13:40:08.917443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.917387 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"5ddf7c1c147bbf925c2b8dbfb38b4e9403e1c59a83c815a6b92eb991858acdce"} Jul 06 13:40:08.917443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.917433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"8ebb3d476a2eec5754732db84a7fd9adb4c2d5b3939d8da22282f1a25a7e2b56"} Jul 06 13:40:08.918852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.918817 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ptzd" event={"ID":"fe5c8515-7f08-4442-ba21-c438df4a8849","Type":"ContainerStarted","Data":"c369006a65d480d69b00f32ad48b6e973a47ef89b7b3beddcc2cba9bd5ed82a1"} Jul 06 13:40:08.920376 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.920354 2562 generic.go:356] "Generic (PLEG): container finished" podID="3ee03c06c928493e4823bc059365a1f8" containerID="75139174cb5d1b87ada942b828c40120ffdfe26fd69a469d2c6ecc4c413927f8" exitCode=0 Jul 06 13:40:08.920485 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.920438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"3ee03c06c928493e4823bc059365a1f8","Type":"ContainerDied","Data":"75139174cb5d1b87ada942b828c40120ffdfe26fd69a469d2c6ecc4c413927f8"} Jul 06 13:40:08.922275 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.922257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" event={"ID":"fa434433-f3d7-457d-a191-0e421b7de99f","Type":"ContainerStarted","Data":"206f1cf9f1962980884ab801dd3b2ef6fddc30c4bf3de9c40eb502a0f9cae8ad"} Jul 06 13:40:08.947470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.947425 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zpc42" podStartSLOduration=10.67478416 podStartE2EDuration="22.947409387s" totalImagesPullingTime="12.272625227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:08.926975781 +0000 UTC m=+24.676789365" watchObservedRunningTime="2026-07-06 13:40:08.947409387 +0000 UTC m=+24.697222952" Jul 06 13:40:08.993662 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.993541 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5g528" podStartSLOduration=10.676697615 podStartE2EDuration="22.993522884s" totalImagesPullingTime="12.316825269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:08.974054275 +0000 UTC m=+24.723867841" watchObservedRunningTime="2026-07-06 13:40:08.993522884 +0000 UTC m=+24.743336478" Jul 06 13:40:08.993890 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:08.993814 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5ptzd" podStartSLOduration=2.19656488 podStartE2EDuration="22.9938039s" totalImagesPullingTime="20.79723902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:08.99349955 +0000 UTC m=+24.743313116" watchObservedRunningTime="2026-07-06 13:40:08.9938039 +0000 UTC m=+24.743617459" Jul 06 13:40:09.050163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.050113 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8kcd9" podStartSLOduration=2.261227572 podStartE2EDuration="23.050098437s" totalImagesPullingTime="20.788870865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:09.049482998 +0000 UTC m=+24.799296562" watchObservedRunningTime="2026-07-06 13:40:09.050098437 +0000 UTC m=+24.799912002" Jul 06 13:40:09.514419 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.514390 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 13:40:09.538216 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.538152 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:09.538362 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.538157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:09.538362 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:09.538279 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:09.538362 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.538302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:09.538522 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:09.538402 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:09.538592 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:09.538554 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:09.927174 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.927134 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" event={"ID":"fa434433-f3d7-457d-a191-0e421b7de99f","Type":"ContainerStarted","Data":"65d6e7a497f2f9ac36ce7a5f339baa2310835ce23f421e99a9227d2f279a003a"} Jul 06 13:40:09.930274 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:09.930207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" event={"ID":"3ee03c06c928493e4823bc059365a1f8","Type":"ContainerStarted","Data":"fbfcf8337d65cd9464a2e7f4639ddb7c92f20ec34b6767fd398c1e2ba589ff15"} Jul 06 13:40:10.415228 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.415132 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T13:40:09.514409069Z","UUID":"edfd4538-4931-4af0-a479-bb4d49528029","Handler":null,"Name":"","Endpoint":""} Jul 06 13:40:10.416744 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.416724 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 13:40:10.416870 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.416766 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 13:40:10.934394 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.934312 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" event={"ID":"fa434433-f3d7-457d-a191-0e421b7de99f","Type":"ContainerStarted","Data":"134a6480d9ac251763f0501936d0920398897fef4e46571d21e381ffd07ce825"} Jul 06 13:40:10.938251 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.938223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"63528f955c482ab58b420bfb75c46bbb20901e2e8a40e3849a6c087b9843db6f"} Jul 06 13:40:10.954155 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.954115 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-202.ec2.internal" podStartSLOduration=24.954102235 podStartE2EDuration="24.954102235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:09.95071371 +0000 UTC m=+25.700527274" watchObservedRunningTime="2026-07-06 13:40:10.954102235 +0000 UTC m=+26.703915796" Jul 06 13:40:10.954332 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:10.954303 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4qtv8" podStartSLOduration=1.95182322 podStartE2EDuration="24.954297821s" totalImagesPullingTime="23.002474601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:10.953521569 +0000 UTC m=+26.703335155" watchObservedRunningTime="2026-07-06 13:40:10.954297821 +0000 UTC m=+26.704111387" Jul 06 13:40:11.538116 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.538082 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:11.538470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.538095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:11.538470 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:11.538214 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:11.538470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.538232 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:11.538470 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:11.538339 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:11.538470 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:11.538422 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:11.728235 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.728195 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5g528" Jul 06 13:40:11.728823 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.728782 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5g528" Jul 06 13:40:11.941464 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.941410 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5g528" Jul 06 13:40:11.942039 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:11.942014 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5g528" Jul 06 13:40:13.537681 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:13.537516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:13.538300 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:13.537518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:13.538300 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:13.537774 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:13.538300 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:13.537524 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:13.538300 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:13.537869 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:13.538300 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:13.537899 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:13.947212 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:13.947172 2562 generic.go:356] "Generic (PLEG): container finished" podID="23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300" containerID="199ac5b694075d53a48ed69a22a5245fd51d8d348c59fbfb32246f0325e5cf1f" exitCode=0 Jul 06 13:40:13.947380 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:13.947261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerDied","Data":"199ac5b694075d53a48ed69a22a5245fd51d8d348c59fbfb32246f0325e5cf1f"} Jul 06 13:40:13.950913 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:13.950890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" event={"ID":"b170b40f-55ae-4016-a4be-f8ca0723f02c","Type":"ContainerStarted","Data":"c8328c7a16e717da23c0292ac4491e5c078ac4e78f48c7bde765c44c87d3bda2"} Jul 06 13:40:14.010895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.010836 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" podStartSLOduration=7.158747924 podStartE2EDuration="28.010815475s" totalImagesPullingTime="20.852067551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:14.009741938 +0000 UTC m=+29.759555526" watchObservedRunningTime="2026-07-06 13:40:14.010815475 +0000 UTC m=+29.760629067" Jul 06 13:40:14.955130 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.954949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerStarted","Data":"aeed644c8aa9a95f7837851d9102d8c2ef26a4194c3c1af0fc1f71767dacb572"} Jul 06 13:40:14.955595 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.955578 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:40:14.955662 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.955639 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:40:14.955716 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.955683 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:40:14.975146 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.975124 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:40:14.985060 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:14.985037 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:40:15.149590 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.149558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8nr9n"] Jul 06 13:40:15.149740 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.149695 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:15.149842 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:15.149818 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:15.150172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.150153 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jsj4h"] Jul 06 13:40:15.150247 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.150236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:15.150343 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:15.150326 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:15.162904 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.162877 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ntd56"] Jul 06 13:40:15.163043 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.162988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:15.163106 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:15.163092 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:15.962330 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.962296 2562 generic.go:356] "Generic (PLEG): container finished" podID="23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300" containerID="aeed644c8aa9a95f7837851d9102d8c2ef26a4194c3c1af0fc1f71767dacb572" exitCode=0 Jul 06 13:40:15.962707 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:15.962369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerDied","Data":"aeed644c8aa9a95f7837851d9102d8c2ef26a4194c3c1af0fc1f71767dacb572"} Jul 06 13:40:16.537916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:16.537880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:16.538097 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:16.538011 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:17.538083 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.537982 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:17.538571 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.538136 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:17.538571 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.538193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:17.538571 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.538314 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:17.662986 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.662957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:17.663134 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.662994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:17.663134 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.663095 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:40:17.663134 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.663112 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 13:40:17.663134 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.663131 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 13:40:17.663263 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.663144 2562 projected.go:196] Error preparing data for projected volume kube-api-access-g7fxg for pod openshift-network-diagnostics/network-check-target-jsj4h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:40:17.663263 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.663145 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs podName:fd5b0edf-c431-4ee8-b16d-27147bc509a4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:49.663131112 +0000 UTC m=+65.412944656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs") pod "network-metrics-daemon-ntd56" (UID: "fd5b0edf-c431-4ee8-b16d-27147bc509a4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 13:40:17.663263 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.663190 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg podName:d4b7fa94-6074-467c-8ae5-a728305229ef nodeName:}" failed. No retries permitted until 2026-07-06 13:40:49.663179364 +0000 UTC m=+65.412992910 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7fxg" (UniqueName: "kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg") pod "network-check-target-jsj4h" (UID: "d4b7fa94-6074-467c-8ae5-a728305229ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 13:40:17.763717 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.763686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:17.763928 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.763816 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 13:40:17.763928 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:17.763864 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret podName:8a3114dd-57c2-4a17-ac11-334b98b10191 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:49.763852753 +0000 UTC m=+65.513666296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret") pod "global-pull-secret-syncer-8nr9n" (UID: "8a3114dd-57c2-4a17-ac11-334b98b10191") : object "kube-system"/"original-pull-secret" not registered Jul 06 13:40:17.968903 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.968868 2562 generic.go:356] "Generic (PLEG): container finished" podID="23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300" containerID="b364f24db0b22ef4df346fb99f7a4459dea1874e5b159dcf798458039c4b3cd6" exitCode=0 Jul 06 13:40:17.969092 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:17.968931 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerDied","Data":"b364f24db0b22ef4df346fb99f7a4459dea1874e5b159dcf798458039c4b3cd6"} Jul 06 13:40:18.537307 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:18.537274 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:18.537487 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:18.537430 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsj4h" podUID="d4b7fa94-6074-467c-8ae5-a728305229ef" Jul 06 13:40:19.538003 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:19.537956 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:19.538477 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:19.538103 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8nr9n" podUID="8a3114dd-57c2-4a17-ac11-334b98b10191" Jul 06 13:40:19.538477 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:19.538161 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:19.538477 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:19.538288 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ntd56" podUID="fd5b0edf-c431-4ee8-b16d-27147bc509a4" Jul 06 13:40:20.391804 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.391756 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-202.ec2.internal" event="NodeReady" Jul 06 13:40:20.391961 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.391901 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 13:40:20.493076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.493047 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ffb96"] Jul 06 13:40:20.495104 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.495082 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.495442 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.495423 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d4mwv"] Jul 06 13:40:20.497030 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.497009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.501539 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.501518 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 13:40:20.501781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.501761 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lqkq5\"" Jul 06 13:40:20.502062 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.502042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 13:40:20.502154 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.502084 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 13:40:20.502284 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.502243 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fhbx9\"" Jul 06 13:40:20.502361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.502297 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 13:40:20.502541 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.502524 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 13:40:20.515463 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.515444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d4mwv"] Jul 06 13:40:20.520614 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.520586 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffb96"] Jul 06 13:40:20.537530 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.537505 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:20.540396 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.540376 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 13:40:20.540761 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.540376 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 13:40:20.540761 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.540562 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rf6qs\"" Jul 06 13:40:20.585806 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.585757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97a078-18f6-4536-bdc5-c5ea3779faaf-tmp-dir\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.585806 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.585801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mct\" (UniqueName: \"kubernetes.io/projected/486b80ea-65a7-4727-bed6-ee1437cb6d36-kube-api-access-65mct\") pod \"ingress-canary-d4mwv\" (UID: \"486b80ea-65a7-4727-bed6-ee1437cb6d36\") " pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.586035 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.585831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/486b80ea-65a7-4727-bed6-ee1437cb6d36-cert\") pod \"ingress-canary-d4mwv\" (UID: \"486b80ea-65a7-4727-bed6-ee1437cb6d36\") " pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.586035 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.585900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df97a078-18f6-4536-bdc5-c5ea3779faaf-config-volume\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.586035 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.585930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4x7\" (UniqueName: \"kubernetes.io/projected/df97a078-18f6-4536-bdc5-c5ea3779faaf-kube-api-access-sk4x7\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.586035 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.585970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.687341 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/486b80ea-65a7-4727-bed6-ee1437cb6d36-cert\") pod \"ingress-canary-d4mwv\" (UID: \"486b80ea-65a7-4727-bed6-ee1437cb6d36\") " pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.687341 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687313 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df97a078-18f6-4536-bdc5-c5ea3779faaf-config-volume\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.687553 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4x7\" (UniqueName: \"kubernetes.io/projected/df97a078-18f6-4536-bdc5-c5ea3779faaf-kube-api-access-sk4x7\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.687553 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.687553 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97a078-18f6-4536-bdc5-c5ea3779faaf-tmp-dir\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.687680 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mct\" (UniqueName: \"kubernetes.io/projected/486b80ea-65a7-4727-bed6-ee1437cb6d36-kube-api-access-65mct\") pod \"ingress-canary-d4mwv\" (UID: \"486b80ea-65a7-4727-bed6-ee1437cb6d36\") " pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.687680 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:20.687665 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:40:20.687764 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:20.687738 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls podName:df97a078-18f6-4536-bdc5-c5ea3779faaf nodeName:}" failed. No retries permitted until 2026-07-06 13:40:21.187718951 +0000 UTC m=+36.937532511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls") pod "dns-default-ffb96" (UID: "df97a078-18f6-4536-bdc5-c5ea3779faaf") : secret "dns-default-metrics-tls" not found Jul 06 13:40:20.687899 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687882 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97a078-18f6-4536-bdc5-c5ea3779faaf-tmp-dir\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.688007 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.687973 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df97a078-18f6-4536-bdc5-c5ea3779faaf-config-volume\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.692621 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.692473 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/486b80ea-65a7-4727-bed6-ee1437cb6d36-cert\") pod \"ingress-canary-d4mwv\" (UID: \"486b80ea-65a7-4727-bed6-ee1437cb6d36\") " pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.704491 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.704462 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4x7\" (UniqueName: \"kubernetes.io/projected/df97a078-18f6-4536-bdc5-c5ea3779faaf-kube-api-access-sk4x7\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:20.704636 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.704605 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mct\" (UniqueName: \"kubernetes.io/projected/486b80ea-65a7-4727-bed6-ee1437cb6d36-kube-api-access-65mct\") pod \"ingress-canary-d4mwv\" (UID: \"486b80ea-65a7-4727-bed6-ee1437cb6d36\") " pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:20.812074 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:20.812028 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d4mwv" Jul 06 13:40:21.191884 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.191842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:21.192082 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:21.192002 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:40:21.192082 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:21.192071 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls podName:df97a078-18f6-4536-bdc5-c5ea3779faaf nodeName:}" failed. No retries permitted until 2026-07-06 13:40:22.192049553 +0000 UTC m=+37.941863116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls") pod "dns-default-ffb96" (UID: "df97a078-18f6-4536-bdc5-c5ea3779faaf") : secret "dns-default-metrics-tls" not found Jul 06 13:40:21.544979 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.544953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:21.545412 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.544957 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:21.550029 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.549989 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 13:40:21.550151 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.549996 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 13:40:21.550939 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.550918 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jzrjm\"" Jul 06 13:40:21.992449 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:21.992358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d4mwv"] Jul 06 13:40:21.996574 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:21.996551 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486b80ea_65a7_4727_bed6_ee1437cb6d36.slice/crio-c377578b309d0cf5ab1d986a6f403902a4ddc69e614ee61c2ef74f13467d7d06 WatchSource:0}: Error finding container c377578b309d0cf5ab1d986a6f403902a4ddc69e614ee61c2ef74f13467d7d06: Status 404 returned error can't find the container with id c377578b309d0cf5ab1d986a6f403902a4ddc69e614ee61c2ef74f13467d7d06 Jul 06 13:40:22.200768 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:22.200675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:22.200942 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:22.200834 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:40:22.200942 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:22.200899 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls podName:df97a078-18f6-4536-bdc5-c5ea3779faaf nodeName:}" failed. No retries permitted until 2026-07-06 13:40:24.200883978 +0000 UTC m=+39.950697520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls") pod "dns-default-ffb96" (UID: "df97a078-18f6-4536-bdc5-c5ea3779faaf") : secret "dns-default-metrics-tls" not found Jul 06 13:40:22.982032 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:22.981817 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d4mwv" event={"ID":"486b80ea-65a7-4727-bed6-ee1437cb6d36","Type":"ContainerStarted","Data":"c377578b309d0cf5ab1d986a6f403902a4ddc69e614ee61c2ef74f13467d7d06"} Jul 06 13:40:22.985055 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:22.985018 2562 generic.go:356] "Generic (PLEG): container finished" podID="23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300" containerID="7dfb6e5862d7278a7154301ce790b1de0089fa6213c677473eee25bcf4249323" exitCode=0 Jul 06 13:40:22.985188 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:22.985074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerDied","Data":"7dfb6e5862d7278a7154301ce790b1de0089fa6213c677473eee25bcf4249323"} Jul 06 13:40:23.551982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:23.551927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:23.647163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:23.647126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:23.992283 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:23.992111 2562 generic.go:356] "Generic (PLEG): container finished" podID="23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300" containerID="e36f85e75a015724f80d25493de94a1d06b7af2491aa44131c18a4e4c8c37dc0" exitCode=0 Jul 06 13:40:23.992283 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:23.992176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerDied","Data":"e36f85e75a015724f80d25493de94a1d06b7af2491aa44131c18a4e4c8c37dc0"} Jul 06 13:40:24.217804 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.217767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:24.217925 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:24.217879 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:40:24.217972 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:24.217934 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls podName:df97a078-18f6-4536-bdc5-c5ea3779faaf nodeName:}" failed. No retries permitted until 2026-07-06 13:40:28.217917582 +0000 UTC m=+43.967731125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls") pod "dns-default-ffb96" (UID: "df97a078-18f6-4536-bdc5-c5ea3779faaf") : secret "dns-default-metrics-tls" not found Jul 06 13:40:24.270444 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.267466 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr"] Jul 06 13:40:24.296807 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.296753 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr"] Jul 06 13:40:24.296932 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.296895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" Jul 06 13:40:24.300163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.300139 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 13:40:24.300431 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.300415 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:40:24.300779 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.300761 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ksqz9\"" Jul 06 13:40:24.373710 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.373678 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c"] Jul 06 13:40:24.393197 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.393163 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68879b88c5-s6vj8"] Jul 06 13:40:24.393391 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.393302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" Jul 06 13:40:24.396377 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.396329 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-84772\"" Jul 06 13:40:24.408983 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.408961 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c"] Jul 06 13:40:24.409097 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.408991 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68879b88c5-s6vj8"] Jul 06 13:40:24.409134 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.409091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.412606 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.412579 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c858x\"" Jul 06 13:40:24.412728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.412608 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 13:40:24.412728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.412646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 13:40:24.413059 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.413046 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 13:40:24.419740 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.419718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v8cd\" (UniqueName: \"kubernetes.io/projected/6b0747ae-f0a5-49e5-948d-87fd0edf0529-kube-api-access-9v8cd\") pod \"volume-data-source-validator-5d456fdbcd-m5btr\" (UID: \"6b0747ae-f0a5-49e5-948d-87fd0edf0529\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" Jul 06 13:40:24.425366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.425342 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 13:40:24.515648 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.515567 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:24.520584 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-bound-sa-token\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520680 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrs7\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-kube-api-access-rcrs7\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520680 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676mt\" (UniqueName: \"kubernetes.io/projected/17e02c5e-8829-4302-9a19-d701b54904a6-kube-api-access-676mt\") pod \"network-check-source-989f8b8dc-ljm5c\" (UID: \"17e02c5e-8829-4302-9a19-d701b54904a6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" Jul 06 13:40:24.520758 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520758 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-trusted-ca\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520864 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-image-registry-private-configuration\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520896 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-registry-certificates\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520925 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v8cd\" (UniqueName: \"kubernetes.io/projected/6b0747ae-f0a5-49e5-948d-87fd0edf0529-kube-api-access-9v8cd\") pod \"volume-data-source-validator-5d456fdbcd-m5btr\" (UID: \"6b0747ae-f0a5-49e5-948d-87fd0edf0529\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" Jul 06 13:40:24.520925 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/597da4a3-5465-46bc-8341-5d235d519614-ca-trust-extracted\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.520996 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.520935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-installation-pull-secrets\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.531259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.531239 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v8cd\" (UniqueName: \"kubernetes.io/projected/6b0747ae-f0a5-49e5-948d-87fd0edf0529-kube-api-access-9v8cd\") pod \"volume-data-source-validator-5d456fdbcd-m5btr\" (UID: \"6b0747ae-f0a5-49e5-948d-87fd0edf0529\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" Jul 06 13:40:24.605705 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.605673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" Jul 06 13:40:24.619669 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.619638 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:24.621673 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676mt\" (UniqueName: \"kubernetes.io/projected/17e02c5e-8829-4302-9a19-d701b54904a6-kube-api-access-676mt\") pod \"network-check-source-989f8b8dc-ljm5c\" (UID: \"17e02c5e-8829-4302-9a19-d701b54904a6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" Jul 06 13:40:24.621741 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.621741 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-trusted-ca\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.621825 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-image-registry-private-configuration\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.621825 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-registry-certificates\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.621825 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:24.621821 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 13:40:24.621974 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:24.621835 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68879b88c5-s6vj8: secret "image-registry-tls" not found Jul 06 13:40:24.621974 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:24.621893 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls podName:597da4a3-5465-46bc-8341-5d235d519614 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:25.121873678 +0000 UTC m=+40.871687235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls") pod "image-registry-68879b88c5-s6vj8" (UID: "597da4a3-5465-46bc-8341-5d235d519614") : secret "image-registry-tls" not found Jul 06 13:40:24.621974 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/597da4a3-5465-46bc-8341-5d235d519614-ca-trust-extracted\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.622123 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.621973 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-installation-pull-secrets\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.622123 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.622023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-bound-sa-token\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.622123 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.622047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrs7\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-kube-api-access-rcrs7\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.622363 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.622336 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/597da4a3-5465-46bc-8341-5d235d519614-ca-trust-extracted\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.622464 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.622435 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-registry-certificates\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.622719 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.622700 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-trusted-ca\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.624364 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.624346 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-installation-pull-secrets\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.624519 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.624497 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-image-registry-private-configuration\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.640220 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.640191 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrs7\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-kube-api-access-rcrs7\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.640525 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.640508 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676mt\" (UniqueName: \"kubernetes.io/projected/17e02c5e-8829-4302-9a19-d701b54904a6-kube-api-access-676mt\") pod \"network-check-source-989f8b8dc-ljm5c\" (UID: \"17e02c5e-8829-4302-9a19-d701b54904a6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" Jul 06 13:40:24.643431 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.643409 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-bound-sa-token\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:24.703686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.703535 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" Jul 06 13:40:24.762329 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.762298 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr"] Jul 06 13:40:24.765232 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:24.765201 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0747ae_f0a5_49e5_948d_87fd0edf0529.slice/crio-408bd30eae2ea63e7beeebc966526a3102c1862b0dc99d8afaefa239878a2cee WatchSource:0}: Error finding container 408bd30eae2ea63e7beeebc966526a3102c1862b0dc99d8afaefa239878a2cee: Status 404 returned error can't find the container with id 408bd30eae2ea63e7beeebc966526a3102c1862b0dc99d8afaefa239878a2cee Jul 06 13:40:24.883386 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.883359 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c"] Jul 06 13:40:24.886387 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:24.886367 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e02c5e_8829_4302_9a19_d701b54904a6.slice/crio-6c0ed9ae1d7ad4588ca27feb6b258c1913ac53ea9e5ead3eb3de91c78ce7d0ac WatchSource:0}: Error finding container 6c0ed9ae1d7ad4588ca27feb6b258c1913ac53ea9e5ead3eb3de91c78ce7d0ac: Status 404 returned error can't find the container with id 6c0ed9ae1d7ad4588ca27feb6b258c1913ac53ea9e5ead3eb3de91c78ce7d0ac Jul 06 13:40:24.995664 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.995633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" event={"ID":"6b0747ae-f0a5-49e5-948d-87fd0edf0529","Type":"ContainerStarted","Data":"408bd30eae2ea63e7beeebc966526a3102c1862b0dc99d8afaefa239878a2cee"} Jul 06 13:40:24.997000 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.996976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d4mwv" event={"ID":"486b80ea-65a7-4727-bed6-ee1437cb6d36","Type":"ContainerStarted","Data":"0b57d4e342ca8d705fa22e305b26a3bc5fe53218fddee03d815c8fbfdccd4c12"} Jul 06 13:40:24.998051 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:24.998030 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" event={"ID":"17e02c5e-8829-4302-9a19-d701b54904a6","Type":"ContainerStarted","Data":"6c0ed9ae1d7ad4588ca27feb6b258c1913ac53ea9e5ead3eb3de91c78ce7d0ac"} Jul 06 13:40:25.001187 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.001169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p27hl" event={"ID":"23dcdf1e-4fb7-46ef-9ffc-8d5d0ddc2300","Type":"ContainerStarted","Data":"a68121a387215e17802b72bd83e327e29841013bcdc35f80d1e23ea733fec464"} Jul 06 13:40:25.025171 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.025083 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d4mwv" podStartSLOduration=2.815753687 podStartE2EDuration="5.025069035s" totalImagesPullingTime="2.209315348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:25.023869362 +0000 UTC m=+40.773682945" watchObservedRunningTime="2026-07-06 13:40:25.025069035 +0000 UTC m=+40.774882600" Jul 06 13:40:25.060461 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.058649 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p27hl" podStartSLOduration=8.227806924 podStartE2EDuration="39.058622563s" totalImagesPullingTime="30.830815639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:25.057121451 +0000 UTC m=+40.806935037" watchObservedRunningTime="2026-07-06 13:40:25.058622563 +0000 UTC m=+40.808436120" Jul 06 13:40:25.060461 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.058815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:25.126350 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.126320 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:25.126535 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:25.126515 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 13:40:25.126587 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:25.126540 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68879b88c5-s6vj8: secret "image-registry-tls" not found Jul 06 13:40:25.126633 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:25.126597 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls podName:597da4a3-5465-46bc-8341-5d235d519614 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:26.126580069 +0000 UTC m=+41.876393616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls") pod "image-registry-68879b88c5-s6vj8" (UID: "597da4a3-5465-46bc-8341-5d235d519614") : secret "image-registry-tls" not found Jul 06 13:40:25.153933 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.153900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:25.448163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.448119 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:25.549827 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:25.549797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:26.026404 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.026368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:26.122952 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.122910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:26.134921 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.134894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:26.135208 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:26.135189 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 13:40:26.135208 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:26.135210 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68879b88c5-s6vj8: secret "image-registry-tls" not found Jul 06 13:40:26.135335 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:26.135262 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls podName:597da4a3-5465-46bc-8341-5d235d519614 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:28.135243351 +0000 UTC m=+43.885056929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls") pod "image-registry-68879b88c5-s6vj8" (UID: "597da4a3-5465-46bc-8341-5d235d519614") : secret "image-registry-tls" not found Jul 06 13:40:26.222143 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.222054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:26.331128 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.331093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:26.656280 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.656252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:26.759835 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:26.759807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:27.008579 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:27.008496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" event={"ID":"6b0747ae-f0a5-49e5-948d-87fd0edf0529","Type":"ContainerStarted","Data":"62ebe4cb4c4f042753837480e96a397b6de39905aca874cd00042834fcdbe43d"} Jul 06 13:40:27.035690 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:27.035640 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-m5btr" podStartSLOduration=1.178301315 podStartE2EDuration="3.035622425s" totalImagesPullingTime="1.85732111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:27.035251206 +0000 UTC m=+42.785064770" watchObservedRunningTime="2026-07-06 13:40:27.035622425 +0000 UTC m=+42.785435989" Jul 06 13:40:27.146200 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:27.146162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:27.217728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:27.217696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:28.149332 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:28.149288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:28.149816 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:28.149435 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 13:40:28.149816 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:28.149459 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68879b88c5-s6vj8: secret "image-registry-tls" not found Jul 06 13:40:28.149816 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:28.149527 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls podName:597da4a3-5465-46bc-8341-5d235d519614 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:32.149506351 +0000 UTC m=+47.899319910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls") pod "image-registry-68879b88c5-s6vj8" (UID: "597da4a3-5465-46bc-8341-5d235d519614") : secret "image-registry-tls" not found Jul 06 13:40:28.249958 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:28.249915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:28.250154 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:28.250090 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:40:28.250212 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:28.250169 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls podName:df97a078-18f6-4536-bdc5-c5ea3779faaf nodeName:}" failed. No retries permitted until 2026-07-06 13:40:36.250152138 +0000 UTC m=+51.999965685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls") pod "dns-default-ffb96" (UID: "df97a078-18f6-4536-bdc5-c5ea3779faaf") : secret "dns-default-metrics-tls" not found Jul 06 13:40:30.018203 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.018158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" event={"ID":"17e02c5e-8829-4302-9a19-d701b54904a6","Type":"ContainerStarted","Data":"83e2f21eb0caced8a0f1bae3c47429d25eb420500e481aaa1672926d0623d9eb"} Jul 06 13:40:30.090340 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.090286 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-ljm5c" podStartSLOduration=1.893186773 podStartE2EDuration="6.090269696s" totalImagesPullingTime="4.197082923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:30.08852292 +0000 UTC m=+45.838336502" watchObservedRunningTime="2026-07-06 13:40:30.090269696 +0000 UTC m=+45.840083312" Jul 06 13:40:30.225533 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.225500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:30.274978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.274903 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r4n8s"] Jul 06 13:40:30.287321 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.287298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.292580 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.292549 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 13:40:30.292681 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.292620 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 13:40:30.292910 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.292895 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 13:40:30.293252 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.293234 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 13:40:30.297084 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.297067 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbfr8\"" Jul 06 13:40:30.314812 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.314773 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4n8s"] Jul 06 13:40:30.358527 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.358498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:30.464803 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.464761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cb10152-577e-4255-ab0b-5539b0b6f8c4-data-volume\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.464946 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.464819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.464946 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.464888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5k5\" (UniqueName: \"kubernetes.io/projected/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-api-access-xv5k5\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.465014 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.464968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cb10152-577e-4255-ab0b-5539b0b6f8c4-crio-socket\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.465014 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.465005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.509699 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.509669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:30.566274 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.566444 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5k5\" (UniqueName: \"kubernetes.io/projected/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-api-access-xv5k5\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.566444 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:30.566418 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:40:30.566553 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cb10152-577e-4255-ab0b-5539b0b6f8c4-crio-socket\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.566553 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:30.566484 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:31.066463933 +0000 UTC m=+46.816277493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : secret "insights-runtime-extractor-tls" not found Jul 06 13:40:30.566553 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.566684 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cb10152-577e-4255-ab0b-5539b0b6f8c4-data-volume\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.566684 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566582 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cb10152-577e-4255-ab0b-5539b0b6f8c4-crio-socket\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.566684 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:30.566609 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:40:30.566684 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:30.566670 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:31.06665385 +0000 UTC m=+46.816467404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : configmap "kube-rbac-proxy" not found Jul 06 13:40:30.566886 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.566871 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cb10152-577e-4255-ab0b-5539b0b6f8c4-data-volume\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.594422 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.594393 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5k5\" (UniqueName: \"kubernetes.io/projected/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-api-access-xv5k5\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:30.602605 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.602580 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz"] Jul 06 13:40:30.624325 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.624301 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" Jul 06 13:40:30.628127 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.628103 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz"] Jul 06 13:40:30.632811 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.632764 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xlmb9\"" Jul 06 13:40:30.632970 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.632951 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 13:40:30.633039 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.632957 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 13:40:30.637586 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.637569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:30.719881 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.719855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:30.768143 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.768115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgttw\" (UniqueName: \"kubernetes.io/projected/41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93-kube-api-access-mgttw\") pod \"migrator-d57d58cc9-5qddz\" (UID: \"41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" Jul 06 13:40:30.868725 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.868654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgttw\" (UniqueName: \"kubernetes.io/projected/41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93-kube-api-access-mgttw\") pod \"migrator-d57d58cc9-5qddz\" (UID: \"41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" Jul 06 13:40:30.873303 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.873274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:30.901119 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.901090 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgttw\" (UniqueName: \"kubernetes.io/projected/41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93-kube-api-access-mgttw\") pod \"migrator-d57d58cc9-5qddz\" (UID: \"41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" Jul 06 13:40:30.934023 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.933989 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" Jul 06 13:40:30.968389 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:30.968366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:31.070850 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:31.070815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:31.071189 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:31.070882 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:40:31.071189 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:31.070953 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:32.070933449 +0000 UTC m=+47.820746991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : secret "insights-runtime-extractor-tls" not found Jul 06 13:40:31.071189 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:31.070997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:31.071189 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:31.071068 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:40:31.071189 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:31.071116 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:32.071101024 +0000 UTC m=+47.820914584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : configmap "kube-rbac-proxy" not found Jul 06 13:40:31.084824 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:31.084756 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz"] Jul 06 13:40:31.086823 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:31.086782 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d31968_0fa2_47f0_b7e2_7f2f7d2e0b93.slice/crio-2b52be170ce66cb9fb88d393e8fe346cd84b9e807c8d31fcb2e52d63a0a7c91c WatchSource:0}: Error finding container 2b52be170ce66cb9fb88d393e8fe346cd84b9e807c8d31fcb2e52d63a0a7c91c: Status 404 returned error can't find the container with id 2b52be170ce66cb9fb88d393e8fe346cd84b9e807c8d31fcb2e52d63a0a7c91c Jul 06 13:40:32.024678 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.024639 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" event={"ID":"41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93","Type":"ContainerStarted","Data":"2b52be170ce66cb9fb88d393e8fe346cd84b9e807c8d31fcb2e52d63a0a7c91c"} Jul 06 13:40:32.078554 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.078514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:32.079009 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.078608 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:32.079009 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.078645 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:40:32.079009 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.078722 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:34.078706195 +0000 UTC m=+49.828519751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : configmap "kube-rbac-proxy" not found Jul 06 13:40:32.079009 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.078737 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:40:32.079009 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.078800 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:34.078766772 +0000 UTC m=+49.828580316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : secret "insights-runtime-extractor-tls" not found Jul 06 13:40:32.179385 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.179347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:32.179568 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.179518 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 13:40:32.179568 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.179540 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68879b88c5-s6vj8: secret "image-registry-tls" not found Jul 06 13:40:32.179676 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:32.179608 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls podName:597da4a3-5465-46bc-8341-5d235d519614 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:40.17958685 +0000 UTC m=+55.929400412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls") pod "image-registry-68879b88c5-s6vj8" (UID: "597da4a3-5465-46bc-8341-5d235d519614") : secret "image-registry-tls" not found Jul 06 13:40:32.445692 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.445663 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpc42_883b955a-f5b4-4cae-87e5-55ea75c2be59/dns-node-resolver/0.log" Jul 06 13:40:32.585223 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.585191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:32.639272 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.639242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:32.753120 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.753049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:32.855106 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:32.855072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:33.027881 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:33.027855 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" event={"ID":"41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93","Type":"ContainerStarted","Data":"6152f9436854f0898954ec50103bff2d5003a82eab472c8b23bd15314e3b5ee0"} Jul 06 13:40:33.041326 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:33.041267 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ptzd_fe5c8515-7f08-4442-ba21-c438df4a8849/node-ca/0.log" Jul 06 13:40:33.412878 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:33.412842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.006587 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.006556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.031822 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.031778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" event={"ID":"41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93","Type":"ContainerStarted","Data":"8bac48a606fe1fdc435fb516644f95ce2ab7441966d969c554abba89864aa25c"} Jul 06 13:40:34.052858 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.052806 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5qddz" podStartSLOduration=2.272501368 podStartE2EDuration="4.052769308s" totalImagesPullingTime="1.78026794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:34.052127741 +0000 UTC m=+49.801941306" watchObservedRunningTime="2026-07-06 13:40:34.052769308 +0000 UTC m=+49.802582872" Jul 06 13:40:34.093587 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.093516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:34.093736 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.093625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:34.093736 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:34.093639 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 13:40:34.093736 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:34.093710 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:38.0936942 +0000 UTC m=+53.843507746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : configmap "kube-rbac-proxy" not found Jul 06 13:40:34.093736 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:34.093729 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:40:34.093918 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:34.093776 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:38.093761773 +0000 UTC m=+53.843575317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : secret "insights-runtime-extractor-tls" not found Jul 06 13:40:34.197237 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.197207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.241432 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.241406 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d4mwv_486b80ea-65a7-4727-bed6-ee1437cb6d36/serve-healthcheck-canary/0.log" Jul 06 13:40:34.584654 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.584624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.685046 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.685010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.786960 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.786927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.885814 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.885729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:34.984942 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:34.984910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:35.106661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:35.106628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:35.904966 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:35.904930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:36.311376 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:36.311343 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:36.311539 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:36.311495 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 13:40:36.311579 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:36.311566 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls podName:df97a078-18f6-4536-bdc5-c5ea3779faaf nodeName:}" failed. No retries permitted until 2026-07-06 13:40:52.311549632 +0000 UTC m=+68.061363174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls") pod "dns-default-ffb96" (UID: "df97a078-18f6-4536-bdc5-c5ea3779faaf") : secret "dns-default-metrics-tls" not found Jul 06 13:40:36.314286 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:36.314262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:36.510947 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:36.510914 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:36.900019 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:36.899975 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:37.311041 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:37.311008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:37.414379 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:37.414350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:37.708938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:37.708870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:37.762139 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:37.762108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:37.901270 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:37.901240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:38.126647 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:38.126610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:38.126830 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:38.126662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:38.126830 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:38.126783 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 13:40:38.126903 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:38.126873 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls podName:6cb10152-577e-4255-ab0b-5539b0b6f8c4 nodeName:}" failed. No retries permitted until 2026-07-06 13:40:46.12685825 +0000 UTC m=+61.876671798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4n8s" (UID: "6cb10152-577e-4255-ab0b-5539b0b6f8c4") : secret "insights-runtime-extractor-tls" not found Jul 06 13:40:38.127138 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:38.127119 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cb10152-577e-4255-ab0b-5539b0b6f8c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:38.508259 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:38.508170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:38.564431 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:38.564392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:38.705875 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:38.705842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:39.113188 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:39.113156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:39.303067 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:39.303032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:39.592380 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:39.592346 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:39.702829 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:39.702779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:39.919075 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:39.918959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:40.244388 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:40.244301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:40.246896 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:40.246857 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"image-registry-68879b88c5-s6vj8\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:40.303993 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:40.303948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:40.318068 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:40.318042 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:40.472914 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:40.472882 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68879b88c5-s6vj8"] Jul 06 13:40:40.476375 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:40.476342 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597da4a3_5465_46bc_8341_5d235d519614.slice/crio-2c957081bb18df1ba0d52a8622fca1da9e170e2fb3dd43df4a6902f3a6357239 WatchSource:0}: Error finding container 2c957081bb18df1ba0d52a8622fca1da9e170e2fb3dd43df4a6902f3a6357239: Status 404 returned error can't find the container with id 2c957081bb18df1ba0d52a8622fca1da9e170e2fb3dd43df4a6902f3a6357239 Jul 06 13:40:40.703833 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:40.703770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:41.055646 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.055609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" event={"ID":"597da4a3-5465-46bc-8341-5d235d519614","Type":"ContainerStarted","Data":"56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad"} Jul 06 13:40:41.055838 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.055679 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" event={"ID":"597da4a3-5465-46bc-8341-5d235d519614","Type":"ContainerStarted","Data":"2c957081bb18df1ba0d52a8622fca1da9e170e2fb3dd43df4a6902f3a6357239"} Jul 06 13:40:41.055838 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.055713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:40:41.078397 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.078352 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" podStartSLOduration=17.078338932 podStartE2EDuration="17.078338932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:41.077489541 +0000 UTC m=+56.827303128" watchObservedRunningTime="2026-07-06 13:40:41.078338932 +0000 UTC m=+56.828152497" Jul 06 13:40:41.104041 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.104013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:41.306562 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.306477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:41.705252 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:41.705170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:42.501822 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:42.501772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:42.711567 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:42.711516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:43.106285 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:43.106252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:43.311976 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:43.311916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:43.901612 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:43.901579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:44.304120 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:44.304091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:44.503542 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:44.503507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:44.703959 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:44.703869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.133965 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.133929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.188449 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.188411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:46.190793 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.190760 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cb10152-577e-4255-ab0b-5539b0b6f8c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4n8s\" (UID: \"6cb10152-577e-4255-ab0b-5539b0b6f8c4\") " pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:46.199961 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.199942 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nbfr8\"" Jul 06 13:40:46.207312 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.207298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4n8s" Jul 06 13:40:46.241896 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.241871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.316905 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.316873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.367577 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.367548 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4n8s"] Jul 06 13:40:46.371092 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:46.371061 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb10152_577e_4255_ab0b_5539b0b6f8c4.slice/crio-4851b4af50ddb623ddae27a35d677a85328fc9fef4832904de81eb8055bfcb76 WatchSource:0}: Error finding container 4851b4af50ddb623ddae27a35d677a85328fc9fef4832904de81eb8055bfcb76: Status 404 returned error can't find the container with id 4851b4af50ddb623ddae27a35d677a85328fc9fef4832904de81eb8055bfcb76 Jul 06 13:40:46.489413 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.489321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.636765 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.636737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.820040 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.819995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.886603 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.886568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:46.985594 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:46.985422 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vsnb" Jul 06 13:40:47.062755 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.062709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:47.075904 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.075818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4n8s" event={"ID":"6cb10152-577e-4255-ab0b-5539b0b6f8c4","Type":"ContainerStarted","Data":"4851b4af50ddb623ddae27a35d677a85328fc9fef4832904de81eb8055bfcb76"} Jul 06 13:40:47.157253 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.157135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:47.431322 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.431204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:47.489463 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.489373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:47.749545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.749455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:47.851328 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:47.851292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:48.072069 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:48.072034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:48.235308 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:48.235269 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:48.335424 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:48.335351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:49.048975 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.048933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:49.148755 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.148721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:49.720027 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.719976 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:49.720488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.720118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:49.722881 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.722857 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fxg\" (UniqueName: \"kubernetes.io/projected/d4b7fa94-6074-467c-8ae5-a728305229ef-kube-api-access-g7fxg\") pod \"network-check-target-jsj4h\" (UID: \"d4b7fa94-6074-467c-8ae5-a728305229ef\") " pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:49.723200 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.723175 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 13:40:49.732706 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.732676 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b0edf-c431-4ee8-b16d-27147bc509a4-metrics-certs\") pod \"network-metrics-daemon-ntd56\" (UID: \"fd5b0edf-c431-4ee8-b16d-27147bc509a4\") " pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:49.764560 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.764526 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jzrjm\"" Jul 06 13:40:49.772255 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.772218 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ntd56" Jul 06 13:40:49.820952 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.820906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:49.823767 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.823739 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 13:40:49.834406 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.834380 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a3114dd-57c2-4a17-ac11-334b98b10191-original-pull-secret\") pod \"global-pull-secret-syncer-8nr9n\" (UID: \"8a3114dd-57c2-4a17-ac11-334b98b10191\") " pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:49.951032 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.950997 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rf6qs\"" Jul 06 13:40:49.958459 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:49.958426 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:50.056245 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:50.056208 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8nr9n" Jul 06 13:40:50.257341 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:50.257310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ntd56"] Jul 06 13:40:50.261248 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:50.261204 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5b0edf_c431_4ee8_b16d_27147bc509a4.slice/crio-15284fa6b9ad1e0ab7d766b0c57c7b2aeac072bc5d18793b18553d7b6cff7b08 WatchSource:0}: Error finding container 15284fa6b9ad1e0ab7d766b0c57c7b2aeac072bc5d18793b18553d7b6cff7b08: Status 404 returned error can't find the container with id 15284fa6b9ad1e0ab7d766b0c57c7b2aeac072bc5d18793b18553d7b6cff7b08 Jul 06 13:40:50.265686 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:50.265651 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8nr9n"] Jul 06 13:40:50.293315 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:50.293282 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jsj4h"] Jul 06 13:40:50.296126 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:50.296097 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b7fa94_6074_467c_8ae5_a728305229ef.slice/crio-8d2c5105e287536f3991e2c7ef48cba65f7ef2ac70d8884e00630a33432f1f90 WatchSource:0}: Error finding container 8d2c5105e287536f3991e2c7ef48cba65f7ef2ac70d8884e00630a33432f1f90: Status 404 returned error can't find the container with id 8d2c5105e287536f3991e2c7ef48cba65f7ef2ac70d8884e00630a33432f1f90 Jul 06 13:40:51.093970 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.093931 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8nr9n" event={"ID":"8a3114dd-57c2-4a17-ac11-334b98b10191","Type":"ContainerStarted","Data":"659367a474f91d8f50d4a7ccb74e75b0b8d4a872417ca356681ba6fe3f060479"} Jul 06 13:40:51.095404 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.095376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jsj4h" event={"ID":"d4b7fa94-6074-467c-8ae5-a728305229ef","Type":"ContainerStarted","Data":"c3360f37cfe179f53df2075169825cb4d3f1a8c99acbdda36787f6bc97f06c30"} Jul 06 13:40:51.095535 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.095423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jsj4h" event={"ID":"d4b7fa94-6074-467c-8ae5-a728305229ef","Type":"ContainerStarted","Data":"8d2c5105e287536f3991e2c7ef48cba65f7ef2ac70d8884e00630a33432f1f90"} Jul 06 13:40:51.095535 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.095442 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:40:51.097009 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.096987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4n8s" event={"ID":"6cb10152-577e-4255-ab0b-5539b0b6f8c4","Type":"ContainerStarted","Data":"dcfc4ef15179b559cd8af4232c794c0dbcec1497d84055fb8c2cd948bf4098c2"} Jul 06 13:40:51.097107 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.097029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4n8s" event={"ID":"6cb10152-577e-4255-ab0b-5539b0b6f8c4","Type":"ContainerStarted","Data":"59ffa28a6d9fa7bcbaea2b39d440295b81386dd666367f32ca2b6b580d21a718"} Jul 06 13:40:51.097940 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.097927 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntd56" event={"ID":"fd5b0edf-c431-4ee8-b16d-27147bc509a4","Type":"ContainerStarted","Data":"15284fa6b9ad1e0ab7d766b0c57c7b2aeac072bc5d18793b18553d7b6cff7b08"} Jul 06 13:40:51.114522 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:51.114469 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jsj4h" podStartSLOduration=65.114451062 podStartE2EDuration="1m5.114451062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:51.113282636 +0000 UTC m=+66.863096240" watchObservedRunningTime="2026-07-06 13:40:51.114451062 +0000 UTC m=+66.864264628" Jul 06 13:40:52.103991 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.103932 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntd56" event={"ID":"fd5b0edf-c431-4ee8-b16d-27147bc509a4","Type":"ContainerStarted","Data":"893231790f1d9d964e521912362ca40ad360852b26d045dcb604627c207df6fc"} Jul 06 13:40:52.104426 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.104048 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ntd56" event={"ID":"fd5b0edf-c431-4ee8-b16d-27147bc509a4","Type":"ContainerStarted","Data":"d5b56f1ae630710c739ac393251eb438ed32e5ed77687ac73633342824edf146"} Jul 06 13:40:52.122780 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.122600 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ntd56" podStartSLOduration=65.261118434 podStartE2EDuration="1m6.122579951s" totalImagesPullingTime="861.461517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:52.121365136 +0000 UTC m=+67.871178726" watchObservedRunningTime="2026-07-06 13:40:52.122579951 +0000 UTC m=+67.872393517" Jul 06 13:40:52.343558 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.343516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:52.346196 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.346168 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df97a078-18f6-4536-bdc5-c5ea3779faaf-metrics-tls\") pod \"dns-default-ffb96\" (UID: \"df97a078-18f6-4536-bdc5-c5ea3779faaf\") " pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:52.609504 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.609472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lqkq5\"" Jul 06 13:40:52.618005 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:52.617972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:53.778159 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:53.777917 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffb96"] Jul 06 13:40:53.922931 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:53.922859 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf97a078_18f6_4536_bdc5_c5ea3779faaf.slice/crio-e9951eb0b3fb3f58c499fc0f8098c06f9457960db9532e5e23e44eb313257de4 WatchSource:0}: Error finding container e9951eb0b3fb3f58c499fc0f8098c06f9457960db9532e5e23e44eb313257de4: Status 404 returned error can't find the container with id e9951eb0b3fb3f58c499fc0f8098c06f9457960db9532e5e23e44eb313257de4 Jul 06 13:40:54.111983 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.111950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffb96" event={"ID":"df97a078-18f6-4536-bdc5-c5ea3779faaf","Type":"ContainerStarted","Data":"e9951eb0b3fb3f58c499fc0f8098c06f9457960db9532e5e23e44eb313257de4"} Jul 06 13:40:54.351099 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.351071 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td"] Jul 06 13:40:54.372678 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.372649 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td"] Jul 06 13:40:54.372882 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.372756 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.375642 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.375609 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 13:40:54.376865 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.376835 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 13:40:54.377039 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.376895 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 13:40:54.377107 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.377084 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 13:40:54.377158 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.377132 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-phrw7\"" Jul 06 13:40:54.379856 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.379829 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68879b88c5-s6vj8"] Jul 06 13:40:54.437380 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.437341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:54.461187 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.461148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhf2\" (UniqueName: \"kubernetes.io/projected/59aae72e-99b0-4ab2-8ea4-691a92583edd-kube-api-access-pfhf2\") pod \"managed-serviceaccount-addon-agent-858c559d8-sp6td\" (UID: \"59aae72e-99b0-4ab2-8ea4-691a92583edd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.461371 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.461222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59aae72e-99b0-4ab2-8ea4-691a92583edd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-858c559d8-sp6td\" (UID: \"59aae72e-99b0-4ab2-8ea4-691a92583edd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.530079 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.530043 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:54.562418 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.562383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59aae72e-99b0-4ab2-8ea4-691a92583edd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-858c559d8-sp6td\" (UID: \"59aae72e-99b0-4ab2-8ea4-691a92583edd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.562595 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.562446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhf2\" (UniqueName: \"kubernetes.io/projected/59aae72e-99b0-4ab2-8ea4-691a92583edd-kube-api-access-pfhf2\") pod \"managed-serviceaccount-addon-agent-858c559d8-sp6td\" (UID: \"59aae72e-99b0-4ab2-8ea4-691a92583edd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.565159 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.565135 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59aae72e-99b0-4ab2-8ea4-691a92583edd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-858c559d8-sp6td\" (UID: \"59aae72e-99b0-4ab2-8ea4-691a92583edd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.571583 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.571561 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhf2\" (UniqueName: \"kubernetes.io/projected/59aae72e-99b0-4ab2-8ea4-691a92583edd-kube-api-access-pfhf2\") pod \"managed-serviceaccount-addon-agent-858c559d8-sp6td\" (UID: \"59aae72e-99b0-4ab2-8ea4-691a92583edd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.627225 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.627131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:54.689121 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.689087 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" Jul 06 13:40:54.729820 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:54.729755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:55.054797 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:55.054739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:55.173135 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:55.173092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:55.522459 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:55.522429 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td"] Jul 06 13:40:55.628065 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:55.628003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:55.946268 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:55.946218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:56.082705 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.082671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:56.120485 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.120416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8nr9n" event={"ID":"8a3114dd-57c2-4a17-ac11-334b98b10191","Type":"ContainerStarted","Data":"09c75d609638e567aa2a9b4f647d113a9cfaaecea313958d447f164f7f0239c1"} Jul 06 13:40:56.122965 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.122931 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4n8s" event={"ID":"6cb10152-577e-4255-ab0b-5539b0b6f8c4","Type":"ContainerStarted","Data":"2b3bc79456c4f81b3f7847ed61adc28228eee1cc46f14205a44fc6c5714e37d6"} Jul 06 13:40:56.124228 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.124202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" event={"ID":"59aae72e-99b0-4ab2-8ea4-691a92583edd","Type":"ContainerStarted","Data":"30db6094cc6806253d5516467b5b3583fe156f084bfb1b8715c81df07974654a"} Jul 06 13:40:56.137239 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.137166 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8nr9n" podStartSLOduration=64.653162834 podStartE2EDuration="1m10.137147634s" totalImagesPullingTime="5.4839848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:56.136779578 +0000 UTC m=+71.886593164" watchObservedRunningTime="2026-07-06 13:40:56.137147634 +0000 UTC m=+71.886961201" Jul 06 13:40:56.156659 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.156595 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r4n8s" podStartSLOduration=17.272219616 podStartE2EDuration="26.156579695s" totalImagesPullingTime="8.884360079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 13:40:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:56.155715973 +0000 UTC m=+71.905529561" watchObservedRunningTime="2026-07-06 13:40:56.156579695 +0000 UTC m=+71.906393260" Jul 06 13:40:56.470518 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.470426 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6bd5ccd-gxc2n"] Jul 06 13:40:56.483315 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.483279 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.483847 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.483558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6bd5ccd-gxc2n"] Jul 06 13:40:56.486702 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.486370 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 13:40:56.486702 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.486677 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 13:40:56.486898 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.486763 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 13:40:56.486970 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.486946 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sc9v9\"" Jul 06 13:40:56.487031 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.487008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 13:40:56.487677 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.487204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 13:40:56.487677 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.487507 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 13:40:56.487677 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.487638 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 13:40:56.577833 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.577762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-service-ca\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.578018 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.577862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-oauth-config\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.578018 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.577941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-serving-cert\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.578018 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.577982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxwj\" (UniqueName: \"kubernetes.io/projected/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-kube-api-access-hhxwj\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.578196 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.578022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-oauth-serving-cert\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.578196 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.578124 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-config\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-config\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679354 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-service-ca\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679354 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-oauth-config\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679354 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-serving-cert\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679354 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxwj\" (UniqueName: \"kubernetes.io/projected/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-kube-api-access-hhxwj\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679354 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-oauth-serving-cert\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.679768 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.679743 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-config\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.680050 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.680027 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-oauth-serving-cert\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.680695 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.680670 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-service-ca\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.686931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.686904 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-serving-cert\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.688365 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.688340 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-oauth-config\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.693724 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.693697 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxwj\" (UniqueName: \"kubernetes.io/projected/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-kube-api-access-hhxwj\") pod \"console-7c6bd5ccd-gxc2n\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:56.798989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:56.798956 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:40:57.459025 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:57.458989 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:57.523142 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:57.523101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:58.253594 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:58.253563 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6bd5ccd-gxc2n"] Jul 06 13:40:58.257673 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:40:58.257640 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1b8863_8f9c_4e74_b6b3_9cd31a6102db.slice/crio-e719c892df0a5e24dfd72c7ee445a2fb03d3abbf3f7d5b0d42be904862c6dffd WatchSource:0}: Error finding container e719c892df0a5e24dfd72c7ee445a2fb03d3abbf3f7d5b0d42be904862c6dffd: Status 404 returned error can't find the container with id e719c892df0a5e24dfd72c7ee445a2fb03d3abbf3f7d5b0d42be904862c6dffd Jul 06 13:40:59.143828 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.143742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bd5ccd-gxc2n" event={"ID":"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db","Type":"ContainerStarted","Data":"e719c892df0a5e24dfd72c7ee445a2fb03d3abbf3f7d5b0d42be904862c6dffd"} Jul 06 13:40:59.145877 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.145810 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffb96" event={"ID":"df97a078-18f6-4536-bdc5-c5ea3779faaf","Type":"ContainerStarted","Data":"94a8ae086a29d46e6e0a505857274c96b3ee67f1bab80ef18128b923e3296544"} Jul 06 13:40:59.145877 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.145868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffb96" event={"ID":"df97a078-18f6-4536-bdc5-c5ea3779faaf","Type":"ContainerStarted","Data":"8b6b9ba7a96be9d7cb1634e6f14431bfaa26909c9e078c82361796639a79bd8c"} Jul 06 13:40:59.146094 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.145900 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ffb96" Jul 06 13:40:59.147318 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.147291 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" event={"ID":"59aae72e-99b0-4ab2-8ea4-691a92583edd","Type":"ContainerStarted","Data":"065d7204953f5da094135c77ca6f9bdfc4143733b54d9e05b9b8fac9ffef8020"} Jul 06 13:40:59.172684 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.172640 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ffb96" podStartSLOduration=35.005146265 podStartE2EDuration="39.17262481s" totalImagesPullingTime="4.167478545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:59.171064632 +0000 UTC m=+74.920878198" watchObservedRunningTime="2026-07-06 13:40:59.17262481 +0000 UTC m=+74.922438374" Jul 06 13:40:59.200809 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.200740 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-858c559d8-sp6td" podStartSLOduration=2.571758847 podStartE2EDuration="5.20072364s" totalImagesPullingTime="2.628964793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:40:59.199173616 +0000 UTC m=+74.948987193" watchObservedRunningTime="2026-07-06 13:40:59.20072364 +0000 UTC m=+74.950537234" Jul 06 13:40:59.451391 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.451284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:59.537915 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.537884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:59.612973 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.612937 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:40:59.632739 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.632702 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h"] Jul 06 13:40:59.635310 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.635268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:40:59.637827 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.637804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 13:40:59.638025 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.638006 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qzdxk\"" Jul 06 13:40:59.644144 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.644120 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h"] Jul 06 13:40:59.709181 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.709093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cl56h\" (UID: \"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:40:59.810006 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:40:59.809965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cl56h\" (UID: \"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:40:59.810180 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:59.810123 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 06 13:40:59.810217 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:40:59.810200 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates podName:b4d3583f-1deb-4d0d-8406-e9d5f28d79e3 nodeName:}" failed. No retries permitted until 2026-07-06 13:41:00.310183301 +0000 UTC m=+76.059996863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-cl56h" (UID: "b4d3583f-1deb-4d0d-8406-e9d5f28d79e3") : secret "prometheus-operator-admission-webhook-tls" not found Jul 06 13:41:00.314409 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:00.314372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cl56h\" (UID: \"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:41:00.314841 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:41:00.314506 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 06 13:41:00.314841 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:41:00.314585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates podName:b4d3583f-1deb-4d0d-8406-e9d5f28d79e3 nodeName:}" failed. No retries permitted until 2026-07-06 13:41:01.314564374 +0000 UTC m=+77.064377931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-cl56h" (UID: "b4d3583f-1deb-4d0d-8406-e9d5f28d79e3") : secret "prometheus-operator-admission-webhook-tls" not found Jul 06 13:41:00.657721 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:00.657629 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:01.320650 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:01.320615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cl56h\" (UID: \"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:41:01.323590 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:01.323561 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b4d3583f-1deb-4d0d-8406-e9d5f28d79e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-cl56h\" (UID: \"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:41:01.446698 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:01.446671 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:41:01.787874 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:01.787841 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h"] Jul 06 13:41:01.792910 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:01.792881 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d3583f_1deb_4d0d_8406_e9d5f28d79e3.slice/crio-34558361f62cd8cf3c6100fa9755bfd66b89b1cb826178651a8b1632c57864b0 WatchSource:0}: Error finding container 34558361f62cd8cf3c6100fa9755bfd66b89b1cb826178651a8b1632c57864b0: Status 404 returned error can't find the container with id 34558361f62cd8cf3c6100fa9755bfd66b89b1cb826178651a8b1632c57864b0 Jul 06 13:41:01.857368 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:01.857336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:01.978962 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:01.978881 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:02.103173 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:02.103143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:02.160166 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:02.160135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" event={"ID":"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3","Type":"ContainerStarted","Data":"34558361f62cd8cf3c6100fa9755bfd66b89b1cb826178651a8b1632c57864b0"} Jul 06 13:41:02.161621 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:02.161592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bd5ccd-gxc2n" event={"ID":"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db","Type":"ContainerStarted","Data":"5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80"} Jul 06 13:41:02.183992 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:02.183920 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c6bd5ccd-gxc2n" podStartSLOduration=2.721088103 podStartE2EDuration="6.183904087s" totalImagesPullingTime="3.462815984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:02.183738492 +0000 UTC m=+77.933552058" watchObservedRunningTime="2026-07-06 13:41:02.183904087 +0000 UTC m=+77.933717652" Jul 06 13:41:02.747037 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:02.746990 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:02.837049 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:02.837013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:04.172798 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.172747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" event={"ID":"b4d3583f-1deb-4d0d-8406-e9d5f28d79e3","Type":"ContainerStarted","Data":"655954f0f6bb4d33bb120f6ed23219d7278f5d4d0d025da0b0217f003295346b"} Jul 06 13:41:04.173180 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.172974 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:41:04.177719 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.177688 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" Jul 06 13:41:04.189486 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.189439 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-cl56h" podStartSLOduration=3.7236089789999998 podStartE2EDuration="5.189424181s" totalImagesPullingTime="1.465815202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:04.188363562 +0000 UTC m=+79.938177143" watchObservedRunningTime="2026-07-06 13:41:04.189424181 +0000 UTC m=+79.939237739" Jul 06 13:41:04.385421 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.385376 2562 patch_prober.go:28] interesting pod/image-registry-68879b88c5-s6vj8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 13:41:04.385570 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.385431 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" podUID="597da4a3-5465-46bc-8341-5d235d519614" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 13:41:04.654435 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.654400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:04.731720 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.731686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:04.874776 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:04.874741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:06.048912 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.048879 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b65779f78-rw2fx"] Jul 06 13:41:06.050845 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.050828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.061895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.061862 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 13:41:06.065726 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.065698 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b65779f78-rw2fx"] Jul 06 13:41:06.155401 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-trusted-ca-bundle\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.155401 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjxh4\" (UniqueName: \"kubernetes.io/projected/f9377816-68fb-46de-b794-37987bde65df-kube-api-access-gjxh4\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.155401 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-oauth-config\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.155401 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-console-config\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.155694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155424 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-oauth-serving-cert\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.155694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-service-ca\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.155694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.155539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-serving-cert\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.254674 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.254640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:06.256688 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.256667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-serving-cert\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.256813 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.256713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-trusted-ca-bundle\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.256813 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.256764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxh4\" (UniqueName: \"kubernetes.io/projected/f9377816-68fb-46de-b794-37987bde65df-kube-api-access-gjxh4\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.256898 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.256826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-oauth-config\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.256898 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.256850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-console-config\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.256991 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.256967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-oauth-serving-cert\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.257139 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.257099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-service-ca\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.257569 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.257545 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-console-config\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.257732 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.257608 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-trusted-ca-bundle\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.257732 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.257651 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-oauth-serving-cert\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.257879 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.257809 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-service-ca\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.259343 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.259320 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-oauth-config\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.259423 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.259330 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-serving-cert\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.265258 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.265239 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxh4\" (UniqueName: \"kubernetes.io/projected/f9377816-68fb-46de-b794-37987bde65df-kube-api-access-gjxh4\") pod \"console-6b65779f78-rw2fx\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.361017 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.360927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:06.484609 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.484574 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b65779f78-rw2fx"] Jul 06 13:41:06.487647 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:06.487617 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9377816_68fb_46de_b794_37987bde65df.slice/crio-51858dec9414e6cbff886d8edfabb0b16d44b4453effd3d9b0dedef5ead546ce WatchSource:0}: Error finding container 51858dec9414e6cbff886d8edfabb0b16d44b4453effd3d9b0dedef5ead546ce: Status 404 returned error can't find the container with id 51858dec9414e6cbff886d8edfabb0b16d44b4453effd3d9b0dedef5ead546ce Jul 06 13:41:06.799968 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.799927 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:41:06.800153 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.799984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:41:06.801341 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.801316 2562 patch_prober.go:28] interesting pod/console-7c6bd5ccd-gxc2n container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" start-of-body= Jul 06 13:41:06.801435 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.801375 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7c6bd5ccd-gxc2n" podUID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" containerName="console" probeResult="failure" output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" Jul 06 13:41:06.999860 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:06.999820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:07.185499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:07.185410 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b65779f78-rw2fx" event={"ID":"f9377816-68fb-46de-b794-37987bde65df","Type":"ContainerStarted","Data":"30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9"} Jul 06 13:41:07.185499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:07.185470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b65779f78-rw2fx" event={"ID":"f9377816-68fb-46de-b794-37987bde65df","Type":"ContainerStarted","Data":"51858dec9414e6cbff886d8edfabb0b16d44b4453effd3d9b0dedef5ead546ce"} Jul 06 13:41:07.205124 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:07.205077 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b65779f78-rw2fx" podStartSLOduration=1.205060733 podStartE2EDuration="1.205060733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:07.203371342 +0000 UTC m=+82.953184928" watchObservedRunningTime="2026-07-06 13:41:07.205060733 +0000 UTC m=+82.954874298" Jul 06 13:41:09.153934 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:09.153905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ffb96" Jul 06 13:41:09.802868 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:09.802832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.020134 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.020086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.060322 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.060248 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7"] Jul 06 13:41:10.062925 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.062909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.065643 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.065620 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 13:41:10.065764 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.065660 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 13:41:10.066745 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.066728 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 13:41:10.066866 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.066850 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8v2g2\"" Jul 06 13:41:10.066866 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.066861 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 13:41:10.066970 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.066874 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 13:41:10.074736 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.074715 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7"] Jul 06 13:41:10.080266 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.080244 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l"] Jul 06 13:41:10.082247 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.082231 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sx2h2"] Jul 06 13:41:10.082402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.082385 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.084127 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.084108 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.085843 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.085826 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 13:41:10.085945 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.085885 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 13:41:10.086509 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.086484 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b2nbk\"" Jul 06 13:41:10.086731 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.086713 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 13:41:10.086937 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.086714 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 13:41:10.087154 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.087135 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 13:41:10.087346 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.087328 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 13:41:10.087968 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.087954 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h5lp7\"" Jul 06 13:41:10.099512 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.099489 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l"] Jul 06 13:41:10.106528 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.106505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.176932 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.176896 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.187997 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.187969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660e9036-f3f6-4056-8c58-b23e624a5bc7-metrics-client-ca\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188079 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-wtmp\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188079 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzgn\" (UniqueName: \"kubernetes.io/projected/fe7f0209-ce35-4970-9545-16905eded10f-kube-api-access-vhzgn\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.188079 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-tls\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188175 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188175 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-sys\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188235 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgj7\" (UniqueName: \"kubernetes.io/projected/0904070f-36bf-4757-951a-35740c0d8c3f-kube-api-access-xlgj7\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.188235 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188235 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0904070f-36bf-4757-951a-35740c0d8c3f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.188324 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.188324 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188310 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0904070f-36bf-4757-951a-35740c0d8c3f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.188389 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0904070f-36bf-4757-951a-35740c0d8c3f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.188389 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.188389 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.188474 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-root\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188474 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe7f0209-ce35-4970-9545-16905eded10f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.188474 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe7f0209-ce35-4970-9545-16905eded10f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.188560 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tfnf\" (UniqueName: \"kubernetes.io/projected/660e9036-f3f6-4056-8c58-b23e624a5bc7-kube-api-access-2tfnf\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.188560 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.188506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-textfile\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.275452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.275420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.289204 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-wtmp\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzgn\" (UniqueName: \"kubernetes.io/projected/fe7f0209-ce35-4970-9545-16905eded10f-kube-api-access-vhzgn\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.289313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-tls\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-sys\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgj7\" (UniqueName: \"kubernetes.io/projected/0904070f-36bf-4757-951a-35740c0d8c3f-kube-api-access-xlgj7\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289336 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-wtmp\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0904070f-36bf-4757-951a-35740c0d8c3f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289425 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-sys\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0904070f-36bf-4757-951a-35740c0d8c3f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.289502 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0904070f-36bf-4757-951a-35740c0d8c3f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-root\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe7f0209-ce35-4970-9545-16905eded10f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289649 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe7f0209-ce35-4970-9545-16905eded10f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tfnf\" (UniqueName: \"kubernetes.io/projected/660e9036-f3f6-4056-8c58-b23e624a5bc7-kube-api-access-2tfnf\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-textfile\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.289984 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.289753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660e9036-f3f6-4056-8c58-b23e624a5bc7-metrics-client-ca\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.290450 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.290431 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/660e9036-f3f6-4056-8c58-b23e624a5bc7-metrics-client-ca\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.290751 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.290727 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.290882 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.290858 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0904070f-36bf-4757-951a-35740c0d8c3f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.291099 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.291076 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/660e9036-f3f6-4056-8c58-b23e624a5bc7-root\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.291176 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.291122 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-textfile\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.291421 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.291390 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe7f0209-ce35-4970-9545-16905eded10f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.291628 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.291588 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe7f0209-ce35-4970-9545-16905eded10f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.292059 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.292036 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.292333 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.292310 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.292460 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.292436 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/660e9036-f3f6-4056-8c58-b23e624a5bc7-node-exporter-tls\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.293080 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.293059 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.293164 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.293147 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0904070f-36bf-4757-951a-35740c0d8c3f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.293217 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.293173 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe7f0209-ce35-4970-9545-16905eded10f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.293804 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.293761 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0904070f-36bf-4757-951a-35740c0d8c3f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.305606 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.305579 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzgn\" (UniqueName: \"kubernetes.io/projected/fe7f0209-ce35-4970-9545-16905eded10f-kube-api-access-vhzgn\") pod \"kube-state-metrics-7c9dbb8b7d-mxl7l\" (UID: \"fe7f0209-ce35-4970-9545-16905eded10f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.305606 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.305604 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tfnf\" (UniqueName: \"kubernetes.io/projected/660e9036-f3f6-4056-8c58-b23e624a5bc7-kube-api-access-2tfnf\") pod \"node-exporter-sx2h2\" (UID: \"660e9036-f3f6-4056-8c58-b23e624a5bc7\") " pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.305799 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.305765 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgj7\" (UniqueName: \"kubernetes.io/projected/0904070f-36bf-4757-951a-35740c0d8c3f-kube-api-access-xlgj7\") pod \"openshift-state-metrics-54cf96dfcf-llfw7\" (UID: \"0904070f-36bf-4757-951a-35740c0d8c3f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.372960 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.372869 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" Jul 06 13:41:10.375054 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.374991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.393015 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.392992 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" Jul 06 13:41:10.397922 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.397898 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sx2h2" Jul 06 13:41:10.408746 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:10.408719 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660e9036_f3f6_4056_8c58_b23e624a5bc7.slice/crio-6c9c1c74d656f29f2471c9ffcbfe09534c421bb344dac31817c3aabe1879ed6e WatchSource:0}: Error finding container 6c9c1c74d656f29f2471c9ffcbfe09534c421bb344dac31817c3aabe1879ed6e: Status 404 returned error can't find the container with id 6c9c1c74d656f29f2471c9ffcbfe09534c421bb344dac31817c3aabe1879ed6e Jul 06 13:41:10.473423 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.473350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.511906 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.511876 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7"] Jul 06 13:41:10.516200 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:10.516166 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0904070f_36bf_4757_951a_35740c0d8c3f.slice/crio-bb160bec1e4416e8e61c22d321e91338bc365cadac0685978c384b15837cc34c WatchSource:0}: Error finding container bb160bec1e4416e8e61c22d321e91338bc365cadac0685978c384b15837cc34c: Status 404 returned error can't find the container with id bb160bec1e4416e8e61c22d321e91338bc365cadac0685978c384b15837cc34c Jul 06 13:41:10.535134 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.535003 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l"] Jul 06 13:41:10.537735 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.537711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.550974 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:10.550947 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7f0209_ce35_4970_9545_16905eded10f.slice/crio-51a144bc2efe3f1b736f1d7b85ffe560470d34d6cf9d0d983a5a56eb9c2d4d20 WatchSource:0}: Error finding container 51a144bc2efe3f1b736f1d7b85ffe560470d34d6cf9d0d983a5a56eb9c2d4d20: Status 404 returned error can't find the container with id 51a144bc2efe3f1b736f1d7b85ffe560470d34d6cf9d0d983a5a56eb9c2d4d20 Jul 06 13:41:10.682273 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.682242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:10.782955 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:10.782920 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:11.122209 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.122172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:11.210885 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.210843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx2h2" event={"ID":"660e9036-f3f6-4056-8c58-b23e624a5bc7","Type":"ContainerStarted","Data":"6c9c1c74d656f29f2471c9ffcbfe09534c421bb344dac31817c3aabe1879ed6e"} Jul 06 13:41:11.212548 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.212476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" event={"ID":"fe7f0209-ce35-4970-9545-16905eded10f","Type":"ContainerStarted","Data":"51a144bc2efe3f1b736f1d7b85ffe560470d34d6cf9d0d983a5a56eb9c2d4d20"} Jul 06 13:41:11.214941 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.214913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" event={"ID":"0904070f-36bf-4757-951a-35740c0d8c3f","Type":"ContainerStarted","Data":"0d97d3518db129167b8fb19c4e30ed2eca13a56372c6be921d497b4fdc8331ed"} Jul 06 13:41:11.215074 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.214970 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" event={"ID":"0904070f-36bf-4757-951a-35740c0d8c3f","Type":"ContainerStarted","Data":"963d76f015f5ea5200d09a267ae350af2e57ffdeffcc018071135ca261f7e2c5"} Jul 06 13:41:11.215074 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.214998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" event={"ID":"0904070f-36bf-4757-951a-35740c0d8c3f","Type":"ContainerStarted","Data":"bb160bec1e4416e8e61c22d321e91338bc365cadac0685978c384b15837cc34c"} Jul 06 13:41:11.294323 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.294282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:11.330999 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.330947 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:41:11.334335 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.334286 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.339658 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339486 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 13:41:11.339658 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339497 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 13:41:11.339658 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339607 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 13:41:11.339891 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 13:41:11.339891 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339776 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 13:41:11.339891 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339850 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 13:41:11.340037 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.339936 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dc876\"" Jul 06 13:41:11.340251 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.340221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 13:41:11.340333 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.340221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 13:41:11.340383 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.340280 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 13:41:11.377116 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.376944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:41:11.407693 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.407653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:11.500975 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.500936 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/095d13b6-fbd4-44f4-98f5-420fd9a17caa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501143 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.500982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/095d13b6-fbd4-44f4-98f5-420fd9a17caa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501143 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/095d13b6-fbd4-44f4-98f5-420fd9a17caa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501143 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501143 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxvg\" (UniqueName: \"kubernetes.io/projected/095d13b6-fbd4-44f4-98f5-420fd9a17caa-kube-api-access-xqxvg\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-web-config\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/095d13b6-fbd4-44f4-98f5-420fd9a17caa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-config-volume\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501655 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/095d13b6-fbd4-44f4-98f5-420fd9a17caa-config-out\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.501655 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.501405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.595964 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.595929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:11.602358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602295 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-web-config\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/095d13b6-fbd4-44f4-98f5-420fd9a17caa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602358 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-config-volume\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/095d13b6-fbd4-44f4-98f5-420fd9a17caa-config-out\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/095d13b6-fbd4-44f4-98f5-420fd9a17caa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/095d13b6-fbd4-44f4-98f5-420fd9a17caa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.602759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/095d13b6-fbd4-44f4-98f5-420fd9a17caa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.603504 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.603504 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.602852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxvg\" (UniqueName: \"kubernetes.io/projected/095d13b6-fbd4-44f4-98f5-420fd9a17caa-kube-api-access-xqxvg\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.604410 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.604044 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/095d13b6-fbd4-44f4-98f5-420fd9a17caa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.604561 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.604436 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/095d13b6-fbd4-44f4-98f5-420fd9a17caa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.606415 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.606370 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/095d13b6-fbd4-44f4-98f5-420fd9a17caa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.607368 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.607320 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-web-config\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.607902 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.607858 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.608111 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.608088 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/095d13b6-fbd4-44f4-98f5-420fd9a17caa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.610052 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.609695 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.610052 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.610012 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.610526 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.610385 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.610526 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.610474 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-config-volume\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.610526 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.610516 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/095d13b6-fbd4-44f4-98f5-420fd9a17caa-config-out\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.611034 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.611009 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/095d13b6-fbd4-44f4-98f5-420fd9a17caa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.615386 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.615363 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxvg\" (UniqueName: \"kubernetes.io/projected/095d13b6-fbd4-44f4-98f5-420fd9a17caa-kube-api-access-xqxvg\") pod \"alertmanager-main-0\" (UID: \"095d13b6-fbd4-44f4-98f5-420fd9a17caa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.647520 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.647401 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 13:41:11.691626 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:11.691590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:12.467420 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:12.467387 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 13:41:12.764950 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:12.764915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:12.868375 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:12.868336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:12.966465 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:12.966434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:13.073913 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.073881 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:13.174837 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.174746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:13.225248 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.225210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" event={"ID":"fe7f0209-ce35-4970-9545-16905eded10f","Type":"ContainerStarted","Data":"46a9bcb5ab4d9f64314d1c6cf85c9f7fa80721a22dc2e633f7d475d04ed3b0b5"} Jul 06 13:41:13.225427 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.225325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" event={"ID":"fe7f0209-ce35-4970-9545-16905eded10f","Type":"ContainerStarted","Data":"a3c68e821c2b95504d39ba7c67fd19233d50664f1ce6248672e45ce615cfe1dd"} Jul 06 13:41:13.225427 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.225362 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" event={"ID":"fe7f0209-ce35-4970-9545-16905eded10f","Type":"ContainerStarted","Data":"4d41916e17318d2f277a84bc559f08920abf11f82a90d502f6c4cab3aba300af"} Jul 06 13:41:13.227853 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.227823 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" event={"ID":"0904070f-36bf-4757-951a-35740c0d8c3f","Type":"ContainerStarted","Data":"4e157f8c03d10dabc89624eedb03798731c7f841beaa4cc49cf4e2a051f74d4c"} Jul 06 13:41:13.229526 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.229503 2562 generic.go:356] "Generic (PLEG): container finished" podID="660e9036-f3f6-4056-8c58-b23e624a5bc7" containerID="30ecfd6ec4dc5ca20118cc4f44ab95e5d200efdce65af17b86cb769e192e0471" exitCode=0 Jul 06 13:41:13.229622 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.229530 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx2h2" event={"ID":"660e9036-f3f6-4056-8c58-b23e624a5bc7","Type":"ContainerDied","Data":"30ecfd6ec4dc5ca20118cc4f44ab95e5d200efdce65af17b86cb769e192e0471"} Jul 06 13:41:13.231370 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.231349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"1b1cebf82d6aeaaebfbee43040d7286708e070b91d9373f64f8e2e234d578f2f"} Jul 06 13:41:13.256342 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.256295 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-mxl7l" podStartSLOduration=1.505339385 podStartE2EDuration="3.256278131s" totalImagesPullingTime="1.750938746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:13.25522773 +0000 UTC m=+89.005041295" watchObservedRunningTime="2026-07-06 13:41:13.256278131 +0000 UTC m=+89.006091695" Jul 06 13:41:13.267763 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.267740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:13.290056 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.289738 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-llfw7" podStartSLOduration=1.6376386790000002 podStartE2EDuration="3.2897155s" totalImagesPullingTime="1.652076821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:13.288610691 +0000 UTC m=+89.038424280" watchObservedRunningTime="2026-07-06 13:41:13.2897155 +0000 UTC m=+89.039529067" Jul 06 13:41:13.687720 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:13.687691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:14.238751 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.238714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx2h2" event={"ID":"660e9036-f3f6-4056-8c58-b23e624a5bc7","Type":"ContainerStarted","Data":"2d82785ea80c175f3a13bdcfa50ef349342bb4e723072affb90a7010172bdc75"} Jul 06 13:41:14.238951 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.238779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx2h2" event={"ID":"660e9036-f3f6-4056-8c58-b23e624a5bc7","Type":"ContainerStarted","Data":"d7208c3406e00ceec2bbd6b4a79caae0faa4f28418b5ce89c1e7c1ba23e3288e"} Jul 06 13:41:14.240210 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.240183 2562 generic.go:356] "Generic (PLEG): container finished" podID="095d13b6-fbd4-44f4-98f5-420fd9a17caa" containerID="7283b02d5126d2b061c63fafd0445cff337b043b8df1bde3415d5ab15ae8f540" exitCode=0 Jul 06 13:41:14.240308 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.240264 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerDied","Data":"7283b02d5126d2b061c63fafd0445cff337b043b8df1bde3415d5ab15ae8f540"} Jul 06 13:41:14.262629 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.262583 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sx2h2" podStartSLOduration=2.2792462159999998 podStartE2EDuration="4.26256659s" totalImagesPullingTime="1.889273379s" totalInitContainerRuntime="94.046995ms" isStatefulPod=true podCreationTimestamp="2026-07-06 13:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:14.261011369 +0000 UTC m=+90.010824956" watchObservedRunningTime="2026-07-06 13:41:14.26256659 +0000 UTC m=+90.012380154" Jul 06 13:41:14.384136 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.384106 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:41:14.829373 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.829339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:14.845661 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.845624 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6bd5ccd-gxc2n"] Jul 06 13:41:14.885460 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.885424 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c87b996bc-9s8g7"] Jul 06 13:41:14.888777 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.888754 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:14.947997 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.947963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c87b996bc-9s8g7"] Jul 06 13:41:14.961462 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:14.961433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:15.036528 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-service-ca\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.036713 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-serving-cert\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.036713 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-config\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.036713 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bx9\" (UniqueName: \"kubernetes.io/projected/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-kube-api-access-c8bx9\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.036713 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-oauth-config\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.036713 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-trusted-ca-bundle\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.036713 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.036709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-oauth-serving-cert\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.058084 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.058051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:15.137685 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bx9\" (UniqueName: \"kubernetes.io/projected/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-kube-api-access-c8bx9\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.137685 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-oauth-config\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.137685 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-trusted-ca-bundle\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.137989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-oauth-serving-cert\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.137989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-service-ca\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.137989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-serving-cert\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.137989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.137777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-config\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.138600 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.138555 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-service-ca\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.138702 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.138679 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-oauth-serving-cert\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.138702 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.138689 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-config\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.138775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.138694 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-trusted-ca-bundle\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.140365 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.140326 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-oauth-config\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.140365 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.140356 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-serving-cert\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.147199 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.147169 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bx9\" (UniqueName: \"kubernetes.io/projected/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-kube-api-access-c8bx9\") pod \"console-c87b996bc-9s8g7\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.200006 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.199968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:15.345017 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.344679 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c87b996bc-9s8g7"] Jul 06 13:41:15.347862 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:15.347829 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fb320f_af44_4c1f_9e5d_2fa1296c1260.slice/crio-cd27f8e11c7b226885bfe6d460241b4a438732268e2c05f269513713e761f99b WatchSource:0}: Error finding container cd27f8e11c7b226885bfe6d460241b4a438732268e2c05f269513713e761f99b: Status 404 returned error can't find the container with id cd27f8e11c7b226885bfe6d460241b4a438732268e2c05f269513713e761f99b Jul 06 13:41:15.373034 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.373002 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d6764b494-v5n8v"] Jul 06 13:41:15.378442 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.378418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.381829 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.381804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 13:41:15.381959 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.381857 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 13:41:15.381959 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.381868 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 13:41:15.381959 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.381893 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-msbc8\"" Jul 06 13:41:15.382177 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.382159 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 13:41:15.386286 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.386266 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 13:41:15.389999 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.389940 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 13:41:15.394595 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.394572 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d6764b494-v5n8v"] Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-metrics-client-ca\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-serving-certs-ca-bundle\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541842 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4slnn\" (UniqueName: \"kubernetes.io/projected/88ada1f9-9094-4535-ab8d-c210fc19b106-kube-api-access-4slnn\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-secret-telemeter-client\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.541976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-federate-client-tls\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.542230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.542021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-telemeter-client-tls\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.642985 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.642888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-secret-telemeter-client\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.642985 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.642949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-federate-client-tls\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.643215 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.642991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-telemeter-client-tls\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.643215 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.643027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-metrics-client-ca\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.643215 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.643070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.643215 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.643100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-serving-certs-ca-bundle\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.643971 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.643937 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-metrics-client-ca\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.643971 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.643949 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-serving-certs-ca-bundle\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.644156 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.643124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.644156 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.644051 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ada1f9-9094-4535-ab8d-c210fc19b106-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.644156 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.644124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4slnn\" (UniqueName: \"kubernetes.io/projected/88ada1f9-9094-4535-ab8d-c210fc19b106-kube-api-access-4slnn\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.646051 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.646021 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-secret-telemeter-client\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.646180 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.646087 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-federate-client-tls\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.646180 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.646138 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.646317 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.646201 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88ada1f9-9094-4535-ab8d-c210fc19b106-telemeter-client-tls\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.654574 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.654523 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4slnn\" (UniqueName: \"kubernetes.io/projected/88ada1f9-9094-4535-ab8d-c210fc19b106-kube-api-access-4slnn\") pod \"telemeter-client-7d6764b494-v5n8v\" (UID: \"88ada1f9-9094-4535-ab8d-c210fc19b106\") " pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.711429 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.711388 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" Jul 06 13:41:15.985893 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:15.985862 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d6764b494-v5n8v"] Jul 06 13:41:15.990073 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:15.990049 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ada1f9_9094_4535_ab8d_c210fc19b106.slice/crio-f72582e8b33bd57d4d73aa9a06b7428c2cceb5612cc84ad48fb6ee97898759b6 WatchSource:0}: Error finding container f72582e8b33bd57d4d73aa9a06b7428c2cceb5612cc84ad48fb6ee97898759b6: Status 404 returned error can't find the container with id f72582e8b33bd57d4d73aa9a06b7428c2cceb5612cc84ad48fb6ee97898759b6 Jul 06 13:41:16.250078 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.250041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" event={"ID":"88ada1f9-9094-4535-ab8d-c210fc19b106","Type":"ContainerStarted","Data":"f72582e8b33bd57d4d73aa9a06b7428c2cceb5612cc84ad48fb6ee97898759b6"} Jul 06 13:41:16.251893 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.251861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c87b996bc-9s8g7" event={"ID":"b7fb320f-af44-4c1f-9e5d-2fa1296c1260","Type":"ContainerStarted","Data":"752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5"} Jul 06 13:41:16.252028 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.251911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c87b996bc-9s8g7" event={"ID":"b7fb320f-af44-4c1f-9e5d-2fa1296c1260","Type":"ContainerStarted","Data":"cd27f8e11c7b226885bfe6d460241b4a438732268e2c05f269513713e761f99b"} Jul 06 13:41:16.255328 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.255299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"dec23b6b925405ac0e1d2ec10a63d1e05b4ef1202447e94d1c0af51d18040131"} Jul 06 13:41:16.255441 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.255379 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"12e178101f42b7b5bf4d5aa6d498e92fc7beba1a6337d7284ec27f53866c6557"} Jul 06 13:41:16.255441 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.255423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"529a961898e9b430bab51f9c235d3326903f8594c931f01673cbaf61be75e44d"} Jul 06 13:41:16.255545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.255464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"0258254ad25bedc8bf2df07cc6b0479068eb10b4405717efd32effbea4154ab3"} Jul 06 13:41:16.255545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.255503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"06292bc9aa0f915537a29726dba9cae7369ccaff9a77830ce818706dab88d7d1"} Jul 06 13:41:16.270592 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.270530 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c87b996bc-9s8g7" podStartSLOduration=2.2705130909999998 podStartE2EDuration="2.270513091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:16.270227283 +0000 UTC m=+92.020040870" watchObservedRunningTime="2026-07-06 13:41:16.270513091 +0000 UTC m=+92.020326656" Jul 06 13:41:16.361712 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.361678 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:16.361911 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.361744 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:16.366680 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.366655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:16.478483 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.478401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:16.678192 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.678158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:16.777978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:16.777894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:17.264627 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:17.264590 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:17.746293 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:17.746250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:17.803088 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:17.803050 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:17.895555 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:17.895522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:18.005084 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:18.004996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:18.100569 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:18.100535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:18.266858 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:18.266775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"095d13b6-fbd4-44f4-98f5-420fd9a17caa","Type":"ContainerStarted","Data":"dd009a929eae2e24a7c8f475cff01c942397cc6588724df05fb8f1ec9891ae8f"} Jul 06 13:41:18.300001 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:18.299921 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.44159332 podStartE2EDuration="7.299903979s" totalImagesPullingTime="3.842358931s" totalInitContainerRuntime="15.951728ms" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:18.299132178 +0000 UTC m=+94.048945765" watchObservedRunningTime="2026-07-06 13:41:18.299903979 +0000 UTC m=+94.049717564" Jul 06 13:41:19.272346 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.272307 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" event={"ID":"88ada1f9-9094-4535-ab8d-c210fc19b106","Type":"ContainerStarted","Data":"76eef150c402ea598d0e983f63c306b43eebd725e65a5759f4d6a73752b42abb"} Jul 06 13:41:19.272728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.272392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" event={"ID":"88ada1f9-9094-4535-ab8d-c210fc19b106","Type":"ContainerStarted","Data":"568cd1ae6f268718389d74e6ef3b612011f3a6c16f1e5727d898dd7b5f917edd"} Jul 06 13:41:19.272728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.272413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" event={"ID":"88ada1f9-9094-4535-ab8d-c210fc19b106","Type":"ContainerStarted","Data":"662c9c631e226838d6c98cf221dcc154667a433098f848a49f6c0d0dd7928add"} Jul 06 13:41:19.301431 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.301377 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d6764b494-v5n8v" podStartSLOduration=1.976403422 podStartE2EDuration="4.301361901s" totalImagesPullingTime="2.324958479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:19.299430453 +0000 UTC m=+95.049244040" watchObservedRunningTime="2026-07-06 13:41:19.301361901 +0000 UTC m=+95.051175465" Jul 06 13:41:19.409703 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.409647 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" podUID="597da4a3-5465-46bc-8341-5d235d519614" containerName="registry" containerID="cri-o://56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad" gracePeriod=30 Jul 06 13:41:19.654714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.654691 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:41:19.683052 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-trusted-ca\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683186 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683064 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-installation-pull-secrets\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683186 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683100 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/597da4a3-5465-46bc-8341-5d235d519614-ca-trust-extracted\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683186 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683130 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683186 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683173 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-registry-certificates\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683395 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683209 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-bound-sa-token\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683395 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683275 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-image-registry-private-configuration\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683395 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcrs7\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-kube-api-access-rcrs7\") pod \"597da4a3-5465-46bc-8341-5d235d519614\" (UID: \"597da4a3-5465-46bc-8341-5d235d519614\") " Jul 06 13:41:19.683540 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683491 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:19.683708 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683684 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-trusted-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.683781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.683707 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:19.686927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.686900 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-kube-api-access-rcrs7" (OuterVolumeSpecName: "kube-api-access-rcrs7") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "kube-api-access-rcrs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:41:19.687050 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.686919 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:19.687050 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.686942 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:41:19.687167 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.687060 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:19.687599 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.687574 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:41:19.693136 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.693111 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597da4a3-5465-46bc-8341-5d235d519614-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "597da4a3-5465-46bc-8341-5d235d519614" (UID: "597da4a3-5465-46bc-8341-5d235d519614"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784890 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-image-registry-private-configuration\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784918 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrs7\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-kube-api-access-rcrs7\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784928 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/597da4a3-5465-46bc-8341-5d235d519614-installation-pull-secrets\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784937 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/597da4a3-5465-46bc-8341-5d235d519614-ca-trust-extracted\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784947 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-registry-tls\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784955 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/597da4a3-5465-46bc-8341-5d235d519614-registry-certificates\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:19.784978 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:19.784963 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/597da4a3-5465-46bc-8341-5d235d519614-bound-sa-token\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:20.164699 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.164665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:20.278029 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.277993 2562 generic.go:356] "Generic (PLEG): container finished" podID="597da4a3-5465-46bc-8341-5d235d519614" containerID="56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad" exitCode=0 Jul 06 13:41:20.278458 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.278039 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" Jul 06 13:41:20.278458 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.278081 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" event={"ID":"597da4a3-5465-46bc-8341-5d235d519614","Type":"ContainerDied","Data":"56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad"} Jul 06 13:41:20.278458 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.278148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68879b88c5-s6vj8" event={"ID":"597da4a3-5465-46bc-8341-5d235d519614","Type":"ContainerDied","Data":"2c957081bb18df1ba0d52a8622fca1da9e170e2fb3dd43df4a6902f3a6357239"} Jul 06 13:41:20.278458 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.278172 2562 scope.go:117] "RemoveContainer" containerID="56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad" Jul 06 13:41:20.287826 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.287808 2562 scope.go:117] "RemoveContainer" containerID="56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad" Jul 06 13:41:20.288081 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:41:20.288062 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad\": container with ID starting with 56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad not found: ID does not exist" containerID="56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad" Jul 06 13:41:20.288137 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.288088 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad"} err="failed to get container status \"56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad\": rpc error: code = NotFound desc = could not find container \"56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad\": container with ID starting with 56d430740d3ec6d3513832b07828520cb83c6ee40ff196951bcde1b7c70434ad not found: ID does not exist" Jul 06 13:41:20.301905 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.301883 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68879b88c5-s6vj8"] Jul 06 13:41:20.304439 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.304421 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68879b88c5-s6vj8"] Jul 06 13:41:20.503042 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.502954 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c87b996bc-9s8g7"] Jul 06 13:41:20.535130 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.535103 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc5d7cfb8-9dt4t"] Jul 06 13:41:20.535414 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.535402 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597da4a3-5465-46bc-8341-5d235d519614" containerName="registry" Jul 06 13:41:20.535464 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.535415 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="597da4a3-5465-46bc-8341-5d235d519614" containerName="registry" Jul 06 13:41:20.535518 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.535507 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="597da4a3-5465-46bc-8341-5d235d519614" containerName="registry" Jul 06 13:41:20.540441 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.540419 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.552352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.552332 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc5d7cfb8-9dt4t"] Jul 06 13:41:20.593013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.592978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-serving-cert\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.593175 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.593031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-trusted-ca-bundle\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.593175 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.593075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-oauth-config\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.593175 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.593103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-oauth-serving-cert\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.593302 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.593188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-config\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.593302 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.593239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-service-ca\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.593302 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.593266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/b61b22ac-1d1c-486b-adfd-01049ea99de7-kube-api-access-fl69v\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.693855 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.693820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-service-ca\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.693855 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.693862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/b61b22ac-1d1c-486b-adfd-01049ea99de7-kube-api-access-fl69v\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694082 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.693895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-serving-cert\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694082 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.693926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-trusted-ca-bundle\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694082 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-oauth-config\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694236 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-oauth-serving-cert\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694236 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-config\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694659 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694633 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-service-ca\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694868 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694843 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-trusted-ca-bundle\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694868 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694860 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-oauth-serving-cert\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.694981 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.694890 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-config\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.696330 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.696306 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-serving-cert\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.696489 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.696467 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-oauth-config\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.703844 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.703824 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/b61b22ac-1d1c-486b-adfd-01049ea99de7-kube-api-access-fl69v\") pod \"console-cc5d7cfb8-9dt4t\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.848851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.848809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:20.985954 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:20.985929 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc5d7cfb8-9dt4t"] Jul 06 13:41:20.988343 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:41:20.988316 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61b22ac_1d1c_486b_adfd_01049ea99de7.slice/crio-9375ee18936850d68d8664319a800bc02f48f460e4f0489ed6d45fb6e6e70afd WatchSource:0}: Error finding container 9375ee18936850d68d8664319a800bc02f48f460e4f0489ed6d45fb6e6e70afd: Status 404 returned error can't find the container with id 9375ee18936850d68d8664319a800bc02f48f460e4f0489ed6d45fb6e6e70afd Jul 06 13:41:21.283384 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:21.283280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5d7cfb8-9dt4t" event={"ID":"b61b22ac-1d1c-486b-adfd-01049ea99de7","Type":"ContainerStarted","Data":"264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b"} Jul 06 13:41:21.283384 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:21.283369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5d7cfb8-9dt4t" event={"ID":"b61b22ac-1d1c-486b-adfd-01049ea99de7","Type":"ContainerStarted","Data":"9375ee18936850d68d8664319a800bc02f48f460e4f0489ed6d45fb6e6e70afd"} Jul 06 13:41:21.313779 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:21.313725 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cc5d7cfb8-9dt4t" podStartSLOduration=1.3137084030000001 podStartE2EDuration="1.313708403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:41:21.311259113 +0000 UTC m=+97.061072705" watchObservedRunningTime="2026-07-06 13:41:21.313708403 +0000 UTC m=+97.063521967" Jul 06 13:41:21.543576 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:21.543545 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="597da4a3-5465-46bc-8341-5d235d519614" path="/var/lib/kubelet/pods/597da4a3-5465-46bc-8341-5d235d519614/volumes" Jul 06 13:41:22.106723 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:22.106695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jsj4h" Jul 06 13:41:25.200490 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:25.200454 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:25.628514 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:25.628465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:25.692238 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:25.692201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:28.985884 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:28.985853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:29.091464 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:29.091428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:30.849613 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:30.849571 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:30.850146 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:30.849674 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:30.854591 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:30.854570 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:31.330322 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:31.330291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:41:31.389399 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:31.389365 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b65779f78-rw2fx"] Jul 06 13:41:31.531353 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:31.531315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:31.629940 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:31.629851 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:31.731464 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:31.731431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:34.728519 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:34.728486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:34.784227 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:34.784190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:36.712762 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:36.712720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:36.825615 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:36.825587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:36.898998 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:36.898963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:39.872834 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:39.872747 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c6bd5ccd-gxc2n" podUID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" containerName="console" containerID="cri-o://5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80" gracePeriod=15 Jul 06 13:41:40.118441 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.118416 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:41:40.162089 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162005 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-oauth-serving-cert\") pod \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " Jul 06 13:41:40.162089 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162076 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-service-ca\") pod \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " Jul 06 13:41:40.162314 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162100 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-config\") pod \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " Jul 06 13:41:40.162314 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162137 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhxwj\" (UniqueName: \"kubernetes.io/projected/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-kube-api-access-hhxwj\") pod \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " Jul 06 13:41:40.162314 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-oauth-config\") pod \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " Jul 06 13:41:40.162314 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162304 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-serving-cert\") pod \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\" (UID: \"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db\") " Jul 06 13:41:40.162525 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162464 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" (UID: "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:40.162525 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162489 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" (UID: "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:40.162621 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162522 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-config" (OuterVolumeSpecName: "console-config") pod "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" (UID: "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:40.162714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162694 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:40.162755 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162719 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:40.162755 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.162735 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:40.164487 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.164459 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" (UID: "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:40.164585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.164490 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" (UID: "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:40.164643 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.164627 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-kube-api-access-hhxwj" (OuterVolumeSpecName: "kube-api-access-hhxwj") pod "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" (UID: "2b1b8863-8f9c-4e74-b6b3-9cd31a6102db"). InnerVolumeSpecName "kube-api-access-hhxwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:41:40.263182 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.263149 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxwj\" (UniqueName: \"kubernetes.io/projected/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-kube-api-access-hhxwj\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:40.263182 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.263180 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:40.263182 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.263189 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:40.363590 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.363553 2562 generic.go:356] "Generic (PLEG): container finished" podID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" containerID="5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80" exitCode=0 Jul 06 13:41:40.363750 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.363607 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bd5ccd-gxc2n" Jul 06 13:41:40.363750 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.363642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bd5ccd-gxc2n" event={"ID":"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db","Type":"ContainerDied","Data":"5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80"} Jul 06 13:41:40.363750 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.363692 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bd5ccd-gxc2n" event={"ID":"2b1b8863-8f9c-4e74-b6b3-9cd31a6102db","Type":"ContainerDied","Data":"e719c892df0a5e24dfd72c7ee445a2fb03d3abbf3f7d5b0d42be904862c6dffd"} Jul 06 13:41:40.363750 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.363711 2562 scope.go:117] "RemoveContainer" containerID="5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80" Jul 06 13:41:40.371828 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.371806 2562 scope.go:117] "RemoveContainer" containerID="5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80" Jul 06 13:41:40.372095 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:41:40.372075 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80\": container with ID starting with 5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80 not found: ID does not exist" containerID="5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80" Jul 06 13:41:40.372156 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.372104 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80"} err="failed to get container status \"5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80\": rpc error: code = NotFound desc = could not find container \"5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80\": container with ID starting with 5d3eff1cea7dc5c0c03ccca9ab1c81cca9a71ef7cdc535e8748a91a373c1eb80 not found: ID does not exist" Jul 06 13:41:40.397818 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.397774 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6bd5ccd-gxc2n"] Jul 06 13:41:40.399078 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:40.399057 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6bd5ccd-gxc2n"] Jul 06 13:41:41.543201 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:41.543161 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" path="/var/lib/kubelet/pods/2b1b8863-8f9c-4e74-b6b3-9cd31a6102db/volumes" Jul 06 13:41:45.522749 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.522690 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c87b996bc-9s8g7" podUID="b7fb320f-af44-4c1f-9e5d-2fa1296c1260" containerName="console" containerID="cri-o://752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5" gracePeriod=15 Jul 06 13:41:45.763536 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.763513 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:45.907452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907421 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-service-ca\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-trusted-ca-bundle\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907649 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907478 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-config\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907649 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907522 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-serving-cert\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907731 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-oauth-config\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907759 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8bx9\" (UniqueName: \"kubernetes.io/projected/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-kube-api-access-c8bx9\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907850 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907830 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-oauth-serving-cert\") pod \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\" (UID: \"b7fb320f-af44-4c1f-9e5d-2fa1296c1260\") " Jul 06 13:41:45.907925 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907905 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-config" (OuterVolumeSpecName: "console-config") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:45.908013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.907986 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:45.908013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.908000 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:45.908230 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.908211 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:45.908310 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.908231 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:45.908310 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.908242 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:45.908310 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.908234 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:45.909640 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.909613 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:45.909850 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.909824 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:45.909938 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:45.909849 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-kube-api-access-c8bx9" (OuterVolumeSpecName: "kube-api-access-c8bx9") pod "b7fb320f-af44-4c1f-9e5d-2fa1296c1260" (UID: "b7fb320f-af44-4c1f-9e5d-2fa1296c1260"). InnerVolumeSpecName "kube-api-access-c8bx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:41:46.009164 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.009125 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:46.009164 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.009156 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:46.009164 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.009168 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bx9\" (UniqueName: \"kubernetes.io/projected/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-kube-api-access-c8bx9\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:46.009164 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.009177 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7fb320f-af44-4c1f-9e5d-2fa1296c1260-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:46.388383 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.388348 2562 generic.go:356] "Generic (PLEG): container finished" podID="b7fb320f-af44-4c1f-9e5d-2fa1296c1260" containerID="752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5" exitCode=0 Jul 06 13:41:46.388663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.388401 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c87b996bc-9s8g7" Jul 06 13:41:46.388663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.388430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c87b996bc-9s8g7" event={"ID":"b7fb320f-af44-4c1f-9e5d-2fa1296c1260","Type":"ContainerDied","Data":"752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5"} Jul 06 13:41:46.388663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.388498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c87b996bc-9s8g7" event={"ID":"b7fb320f-af44-4c1f-9e5d-2fa1296c1260","Type":"ContainerDied","Data":"cd27f8e11c7b226885bfe6d460241b4a438732268e2c05f269513713e761f99b"} Jul 06 13:41:46.388663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.388522 2562 scope.go:117] "RemoveContainer" containerID="752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5" Jul 06 13:41:46.398992 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.398976 2562 scope.go:117] "RemoveContainer" containerID="752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5" Jul 06 13:41:46.399263 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:41:46.399242 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5\": container with ID starting with 752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5 not found: ID does not exist" containerID="752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5" Jul 06 13:41:46.399325 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.399276 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5"} err="failed to get container status \"752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5\": rpc error: code = NotFound desc = could not find container \"752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5\": container with ID starting with 752822b9b1f045717d1369ecc01db2bdcb432d0c4e85fbf86efc818ad8ccc7d5 not found: ID does not exist" Jul 06 13:41:46.414018 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.413981 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c87b996bc-9s8g7"] Jul 06 13:41:46.419089 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:46.419068 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c87b996bc-9s8g7"] Jul 06 13:41:47.543242 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:47.543209 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7fb320f-af44-4c1f-9e5d-2fa1296c1260" path="/var/lib/kubelet/pods/b7fb320f-af44-4c1f-9e5d-2fa1296c1260/volumes" Jul 06 13:41:47.748355 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:47.748315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:47.846979 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:47.846893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:41:56.410085 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.410043 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6b65779f78-rw2fx" podUID="f9377816-68fb-46de-b794-37987bde65df" containerName="console" containerID="cri-o://30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9" gracePeriod=15 Jul 06 13:41:56.670070 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.670010 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:56.796980 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.796948 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjxh4\" (UniqueName: \"kubernetes.io/projected/f9377816-68fb-46de-b794-37987bde65df-kube-api-access-gjxh4\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797162 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.796990 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-service-ca\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797162 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797021 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-serving-cert\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797162 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797047 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-oauth-config\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797162 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797104 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-oauth-serving-cert\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797162 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797140 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-console-config\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797396 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797178 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-trusted-ca-bundle\") pod \"f9377816-68fb-46de-b794-37987bde65df\" (UID: \"f9377816-68fb-46de-b794-37987bde65df\") " Jul 06 13:41:56.797646 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797594 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-service-ca" (OuterVolumeSpecName: "service-ca") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:56.797775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797642 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:56.797775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797725 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:56.797775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.797752 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-console-config" (OuterVolumeSpecName: "console-config") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:41:56.799597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.799568 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:56.799597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.799587 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:41:56.799597 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.799576 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9377816-68fb-46de-b794-37987bde65df-kube-api-access-gjxh4" (OuterVolumeSpecName: "kube-api-access-gjxh4") pod "f9377816-68fb-46de-b794-37987bde65df" (UID: "f9377816-68fb-46de-b794-37987bde65df"). InnerVolumeSpecName "kube-api-access-gjxh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:41:56.898452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898407 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjxh4\" (UniqueName: \"kubernetes.io/projected/f9377816-68fb-46de-b794-37987bde65df-kube-api-access-gjxh4\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:56.898452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898449 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:56.898714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898463 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:56.898714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898476 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9377816-68fb-46de-b794-37987bde65df-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:56.898714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898488 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:56.898714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898500 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:56.898714 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:56.898513 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9377816-68fb-46de-b794-37987bde65df-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:41:57.439809 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.436230 2562 generic.go:356] "Generic (PLEG): container finished" podID="f9377816-68fb-46de-b794-37987bde65df" containerID="30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9" exitCode=0 Jul 06 13:41:57.439809 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.436292 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b65779f78-rw2fx" event={"ID":"f9377816-68fb-46de-b794-37987bde65df","Type":"ContainerDied","Data":"30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9"} Jul 06 13:41:57.439809 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.436380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b65779f78-rw2fx" event={"ID":"f9377816-68fb-46de-b794-37987bde65df","Type":"ContainerDied","Data":"51858dec9414e6cbff886d8edfabb0b16d44b4453effd3d9b0dedef5ead546ce"} Jul 06 13:41:57.439809 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.436410 2562 scope.go:117] "RemoveContainer" containerID="30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9" Jul 06 13:41:57.439809 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.436581 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b65779f78-rw2fx" Jul 06 13:41:57.455621 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.455596 2562 scope.go:117] "RemoveContainer" containerID="30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9" Jul 06 13:41:57.455905 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:41:57.455881 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9\": container with ID starting with 30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9 not found: ID does not exist" containerID="30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9" Jul 06 13:41:57.456037 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.455918 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9"} err="failed to get container status \"30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9\": rpc error: code = NotFound desc = could not find container \"30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9\": container with ID starting with 30da992d58d9ebb4f6f4468a5e4ddcb34622f3cf8e06e17c096e55f431706ad9 not found: ID does not exist" Jul 06 13:41:57.481345 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.481313 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b65779f78-rw2fx"] Jul 06 13:41:57.491328 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.491290 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b65779f78-rw2fx"] Jul 06 13:41:57.545779 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:41:57.545737 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9377816-68fb-46de-b794-37987bde65df" path="/var/lib/kubelet/pods/f9377816-68fb-46de-b794-37987bde65df/volumes" Jul 06 13:42:21.636352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:21.636316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:26.253903 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.253872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:26.355647 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.355605 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:26.982716 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.982683 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64786c6f75-hxfh6"] Jul 06 13:42:26.983008 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.982996 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7fb320f-af44-4c1f-9e5d-2fa1296c1260" containerName="console" Jul 06 13:42:26.983053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983010 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7fb320f-af44-4c1f-9e5d-2fa1296c1260" containerName="console" Jul 06 13:42:26.983053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983025 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9377816-68fb-46de-b794-37987bde65df" containerName="console" Jul 06 13:42:26.983053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983031 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9377816-68fb-46de-b794-37987bde65df" containerName="console" Jul 06 13:42:26.983053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983048 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" containerName="console" Jul 06 13:42:26.983053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983053 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" containerName="console" Jul 06 13:42:26.983198 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983102 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1b8863-8f9c-4e74-b6b3-9cd31a6102db" containerName="console" Jul 06 13:42:26.983198 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983114 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7fb320f-af44-4c1f-9e5d-2fa1296c1260" containerName="console" Jul 06 13:42:26.983198 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.983120 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9377816-68fb-46de-b794-37987bde65df" containerName="console" Jul 06 13:42:26.987331 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:26.987314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039275 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-serving-cert\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039275 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-trusted-ca-bundle\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039494 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-service-ca\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039494 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-console-config\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039494 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l28n\" (UniqueName: \"kubernetes.io/projected/a2ac2931-157e-40e9-be54-36f444ecdfef-kube-api-access-9l28n\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039494 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-oauth-config\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.039494 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.039450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-oauth-serving-cert\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.069981 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.069949 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64786c6f75-hxfh6"] Jul 06 13:42:27.080489 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.080448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:27.140145 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-serving-cert\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.140145 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-trusted-ca-bundle\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.140409 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-service-ca\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.140409 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-console-config\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.140521 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l28n\" (UniqueName: \"kubernetes.io/projected/a2ac2931-157e-40e9-be54-36f444ecdfef-kube-api-access-9l28n\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.140521 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-oauth-config\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.140521 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.140493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-oauth-serving-cert\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.141072 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.141044 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-service-ca\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.141072 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.141057 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-trusted-ca-bundle\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.141072 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.141076 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-console-config\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.141240 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.141153 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-oauth-serving-cert\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.143151 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.143133 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-serving-cert\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.143233 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.143166 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-oauth-config\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.157762 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.157730 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l28n\" (UniqueName: \"kubernetes.io/projected/a2ac2931-157e-40e9-be54-36f444ecdfef-kube-api-access-9l28n\") pod \"console-64786c6f75-hxfh6\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.162374 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.162349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:27.251157 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.251075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:27.296094 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.296046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:27.348659 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.348567 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:27.437133 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.437109 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64786c6f75-hxfh6"] Jul 06 13:42:27.439458 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:42:27.439435 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ac2931_157e_40e9_be54_36f444ecdfef.slice/crio-c6a5fbce53dfdc1ebe07cc6d338f523b6aaa7178fff9b0d5db7896f67f93c0f8 WatchSource:0}: Error finding container c6a5fbce53dfdc1ebe07cc6d338f523b6aaa7178fff9b0d5db7896f67f93c0f8: Status 404 returned error can't find the container with id c6a5fbce53dfdc1ebe07cc6d338f523b6aaa7178fff9b0d5db7896f67f93c0f8 Jul 06 13:42:27.554333 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.554298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64786c6f75-hxfh6" event={"ID":"a2ac2931-157e-40e9-be54-36f444ecdfef","Type":"ContainerStarted","Data":"1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863"} Jul 06 13:42:27.554488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.554348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64786c6f75-hxfh6" event={"ID":"a2ac2931-157e-40e9-be54-36f444ecdfef","Type":"ContainerStarted","Data":"c6a5fbce53dfdc1ebe07cc6d338f523b6aaa7178fff9b0d5db7896f67f93c0f8"} Jul 06 13:42:27.581652 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.581596 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64786c6f75-hxfh6" podStartSLOduration=1.581580191 podStartE2EDuration="1.581580191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:42:27.580629909 +0000 UTC m=+163.330443485" watchObservedRunningTime="2026-07-06 13:42:27.581580191 +0000 UTC m=+163.331393756" Jul 06 13:42:27.666749 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.666715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:27.750630 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:27.750599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:37.297172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.297087 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:37.297172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.297148 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:37.301986 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.301960 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:37.595049 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.594957 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:42:37.665104 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.665075 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc5d7cfb8-9dt4t"] Jul 06 13:42:37.778246 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.778204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:37.876705 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.876635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:37.976712 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:37.976678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:38.251217 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:38.251115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:38.352205 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:38.352172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:43.743894 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:43.743851 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:43.832663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:43.832634 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:47.767956 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:47.767916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:47.867976 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:47.867941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:42:47.971136 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:42:47.971108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:02.685298 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:02.685231 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-cc5d7cfb8-9dt4t" podUID="b61b22ac-1d1c-486b-adfd-01049ea99de7" containerName="console" containerID="cri-o://264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b" gracePeriod=15 Jul 06 13:43:02.928177 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:02.928154 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:43:03.048807 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.048758 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-serving-cert\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.048982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.048817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-oauth-config\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.048982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.048840 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/b61b22ac-1d1c-486b-adfd-01049ea99de7-kube-api-access-fl69v\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.048982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.048877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-oauth-serving-cert\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.048982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.048901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-config\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.048982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.048938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-service-ca\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.049235 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.049036 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-trusted-ca-bundle\") pod \"b61b22ac-1d1c-486b-adfd-01049ea99de7\" (UID: \"b61b22ac-1d1c-486b-adfd-01049ea99de7\") " Jul 06 13:43:03.049368 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.049336 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:43:03.049368 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.049332 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-config" (OuterVolumeSpecName: "console-config") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:43:03.049510 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.049467 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-service-ca" (OuterVolumeSpecName: "service-ca") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:43:03.049634 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.049609 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:43:03.051064 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.051039 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:43:03.051064 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.051048 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61b22ac-1d1c-486b-adfd-01049ea99de7-kube-api-access-fl69v" (OuterVolumeSpecName: "kube-api-access-fl69v") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "kube-api-access-fl69v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:43:03.051186 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.051053 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b61b22ac-1d1c-486b-adfd-01049ea99de7" (UID: "b61b22ac-1d1c-486b-adfd-01049ea99de7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:43:03.149814 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149751 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.149814 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149779 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.149814 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149807 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.149814 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149816 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.149814 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149824 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/b61b22ac-1d1c-486b-adfd-01049ea99de7-kube-api-access-fl69v\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.150103 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149833 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.150103 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.149841 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b61b22ac-1d1c-486b-adfd-01049ea99de7-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:03.694692 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.694656 2562 generic.go:356] "Generic (PLEG): container finished" podID="b61b22ac-1d1c-486b-adfd-01049ea99de7" containerID="264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b" exitCode=0 Jul 06 13:43:03.695172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.694704 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5d7cfb8-9dt4t" Jul 06 13:43:03.695172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.694738 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5d7cfb8-9dt4t" event={"ID":"b61b22ac-1d1c-486b-adfd-01049ea99de7","Type":"ContainerDied","Data":"264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b"} Jul 06 13:43:03.695172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.694814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5d7cfb8-9dt4t" event={"ID":"b61b22ac-1d1c-486b-adfd-01049ea99de7","Type":"ContainerDied","Data":"9375ee18936850d68d8664319a800bc02f48f460e4f0489ed6d45fb6e6e70afd"} Jul 06 13:43:03.695172 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.694839 2562 scope.go:117] "RemoveContainer" containerID="264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b" Jul 06 13:43:03.723271 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.723250 2562 scope.go:117] "RemoveContainer" containerID="264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b" Jul 06 13:43:03.723602 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:03.723578 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b\": container with ID starting with 264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b not found: ID does not exist" containerID="264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b" Jul 06 13:43:03.723669 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.723611 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b"} err="failed to get container status \"264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b\": rpc error: code = NotFound desc = could not find container \"264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b\": container with ID starting with 264bbe8387bac73e2d19cb42e2d81521157588de01599a9a42a53fa49cd9924b not found: ID does not exist" Jul 06 13:43:03.735116 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.735059 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc5d7cfb8-9dt4t"] Jul 06 13:43:03.737363 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:03.737337 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cc5d7cfb8-9dt4t"] Jul 06 13:43:05.542860 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:05.542830 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61b22ac-1d1c-486b-adfd-01049ea99de7" path="/var/lib/kubelet/pods/b61b22ac-1d1c-486b-adfd-01049ea99de7/volumes" Jul 06 13:43:10.797167 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:10.797135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:15.774078 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:15.774038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:15.835409 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:15.835379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:17.232828 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.232773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:17.357552 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.357525 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:17.988272 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.988236 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z"] Jul 06 13:43:17.988560 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.988548 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61b22ac-1d1c-486b-adfd-01049ea99de7" containerName="console" Jul 06 13:43:17.988603 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.988564 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61b22ac-1d1c-486b-adfd-01049ea99de7" containerName="console" Jul 06 13:43:17.988636 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.988612 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61b22ac-1d1c-486b-adfd-01049ea99de7" containerName="console" Jul 06 13:43:17.991746 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:17.991729 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:17.997108 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:17.997077 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:17.998733 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:17.998705 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-q4td2\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4td2\"" type="*v1.Secret" Jul 06 13:43:17.998733 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:17.998710 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 13:43:17.998957 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:17.998938 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-134-202.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-134-202.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 13:43:18.009663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.009634 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z"] Jul 06 13:43:18.069849 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.069804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.070026 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.069868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.070026 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.069923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbmp\" (UniqueName: \"kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.171105 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.171070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.171105 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.171108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbmp\" (UniqueName: \"kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.171268 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.171189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.171526 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.171503 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:18.171606 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:18.171519 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:19.121581 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:19.121547 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4td2\"" Jul 06 13:43:19.182145 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:19.182105 2562 projected.go:291] Couldn't get configMap openshift-marketplace/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 06 13:43:19.436060 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:19.435965 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 13:43:19.540125 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:19.540094 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 13:43:19.542429 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:19.542410 2562 projected.go:196] Error preparing data for projected volume kube-api-access-bsbmp for pod openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z: failed to sync configmap cache: timed out waiting for the condition Jul 06 13:43:19.542518 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:43:19.542489 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp podName:106ea5e8-04f8-4387-a325-922f0c746baf nodeName:}" failed. No retries permitted until 2026-07-06 13:43:20.042473098 +0000 UTC m=+215.792286655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bsbmp" (UniqueName: "kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp") pod "685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" (UID: "106ea5e8-04f8-4387-a325-922f0c746baf") : failed to sync configmap cache: timed out waiting for the condition Jul 06 13:43:20.088127 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:20.088088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbmp\" (UniqueName: \"kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:20.090720 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:20.090696 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbmp\" (UniqueName: \"kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:20.099830 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:20.099808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:20.239216 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:20.239191 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z"] Jul 06 13:43:20.241415 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:43:20.241385 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106ea5e8_04f8_4387_a325_922f0c746baf.slice/crio-962af9c4e76b9eea22fb2a0aafff21e4600930a497aae93a44bdd0fc22e44e28 WatchSource:0}: Error finding container 962af9c4e76b9eea22fb2a0aafff21e4600930a497aae93a44bdd0fc22e44e28: Status 404 returned error can't find the container with id 962af9c4e76b9eea22fb2a0aafff21e4600930a497aae93a44bdd0fc22e44e28 Jul 06 13:43:20.760398 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:20.760363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" event={"ID":"106ea5e8-04f8-4387-a325-922f0c746baf","Type":"ContainerStarted","Data":"962af9c4e76b9eea22fb2a0aafff21e4600930a497aae93a44bdd0fc22e44e28"} Jul 06 13:43:27.789093 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:27.789060 2562 generic.go:356] "Generic (PLEG): container finished" podID="106ea5e8-04f8-4387-a325-922f0c746baf" containerID="c79393d39e245928f0c8dfb9d4e5b93af1e1a68197b9734b3f538b9414226a9b" exitCode=0 Jul 06 13:43:27.789465 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:27.789140 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" event={"ID":"106ea5e8-04f8-4387-a325-922f0c746baf","Type":"ContainerDied","Data":"c79393d39e245928f0c8dfb9d4e5b93af1e1a68197b9734b3f538b9414226a9b"} Jul 06 13:43:30.802455 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:30.802419 2562 generic.go:356] "Generic (PLEG): container finished" podID="106ea5e8-04f8-4387-a325-922f0c746baf" containerID="5fad215733de496f3db29bfb36356f6c6342822ada6284b90a23b7258ae923ae" exitCode=0 Jul 06 13:43:30.802849 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:30.802506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" event={"ID":"106ea5e8-04f8-4387-a325-922f0c746baf","Type":"ContainerDied","Data":"5fad215733de496f3db29bfb36356f6c6342822ada6284b90a23b7258ae923ae"} Jul 06 13:43:38.839860 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:38.839822 2562 generic.go:356] "Generic (PLEG): container finished" podID="106ea5e8-04f8-4387-a325-922f0c746baf" containerID="efb9cdfa64d864682941c3e255a9faa40bd375174db2b0bbd506d1c5e086f2de" exitCode=0 Jul 06 13:43:38.840223 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:38.839899 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" event={"ID":"106ea5e8-04f8-4387-a325-922f0c746baf","Type":"ContainerDied","Data":"efb9cdfa64d864682941c3e255a9faa40bd375174db2b0bbd506d1c5e086f2de"} Jul 06 13:43:39.972156 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:39.972131 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:40.073565 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.073521 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsbmp\" (UniqueName: \"kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp\") pod \"106ea5e8-04f8-4387-a325-922f0c746baf\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " Jul 06 13:43:40.073760 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.073584 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-bundle\") pod \"106ea5e8-04f8-4387-a325-922f0c746baf\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " Jul 06 13:43:40.073760 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.073724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-util\") pod \"106ea5e8-04f8-4387-a325-922f0c746baf\" (UID: \"106ea5e8-04f8-4387-a325-922f0c746baf\") " Jul 06 13:43:40.074452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.074424 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-bundle" (OuterVolumeSpecName: "bundle") pod "106ea5e8-04f8-4387-a325-922f0c746baf" (UID: "106ea5e8-04f8-4387-a325-922f0c746baf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:43:40.075688 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.075666 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp" (OuterVolumeSpecName: "kube-api-access-bsbmp") pod "106ea5e8-04f8-4387-a325-922f0c746baf" (UID: "106ea5e8-04f8-4387-a325-922f0c746baf"). InnerVolumeSpecName "kube-api-access-bsbmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:43:40.079628 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.079600 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-util" (OuterVolumeSpecName: "util") pod "106ea5e8-04f8-4387-a325-922f0c746baf" (UID: "106ea5e8-04f8-4387-a325-922f0c746baf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:43:40.175209 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.175112 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:40.175209 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.175147 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbmp\" (UniqueName: \"kubernetes.io/projected/106ea5e8-04f8-4387-a325-922f0c746baf-kube-api-access-bsbmp\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:40.175209 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.175157 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/106ea5e8-04f8-4387-a325-922f0c746baf-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:40.850013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.849978 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" event={"ID":"106ea5e8-04f8-4387-a325-922f0c746baf","Type":"ContainerDied","Data":"962af9c4e76b9eea22fb2a0aafff21e4600930a497aae93a44bdd0fc22e44e28"} Jul 06 13:43:40.850226 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.850035 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d2q9z" Jul 06 13:43:40.850226 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:40.850050 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962af9c4e76b9eea22fb2a0aafff21e4600930a497aae93a44bdd0fc22e44e28" Jul 06 13:43:44.036985 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:44.036955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:45.070549 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:45.070510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:46.068002 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.067969 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf"] Jul 06 13:43:46.068300 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068288 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="util" Jul 06 13:43:46.068352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068302 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="util" Jul 06 13:43:46.068352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068310 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="pull" Jul 06 13:43:46.068352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068316 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="pull" Jul 06 13:43:46.068352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068325 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="extract" Jul 06 13:43:46.068352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068332 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="extract" Jul 06 13:43:46.068491 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.068377 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="106ea5e8-04f8-4387-a325-922f0c746baf" containerName="extract" Jul 06 13:43:46.099583 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.099558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf"] Jul 06 13:43:46.100005 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.099649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.102654 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.102631 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 13:43:46.102781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.102687 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:43:46.102781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.102693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8pb5c\"" Jul 06 13:43:46.227064 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.227032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/585f71b9-5274-410c-9023-d7d69b2a90da-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-52rcf\" (UID: \"585f71b9-5274-410c-9023-d7d69b2a90da\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.227256 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.227080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9b5\" (UniqueName: \"kubernetes.io/projected/585f71b9-5274-410c-9023-d7d69b2a90da-kube-api-access-7x9b5\") pod \"cert-manager-operator-controller-manager-55c444bf86-52rcf\" (UID: \"585f71b9-5274-410c-9023-d7d69b2a90da\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.328160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.328071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/585f71b9-5274-410c-9023-d7d69b2a90da-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-52rcf\" (UID: \"585f71b9-5274-410c-9023-d7d69b2a90da\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.328160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.328125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9b5\" (UniqueName: \"kubernetes.io/projected/585f71b9-5274-410c-9023-d7d69b2a90da-kube-api-access-7x9b5\") pod \"cert-manager-operator-controller-manager-55c444bf86-52rcf\" (UID: \"585f71b9-5274-410c-9023-d7d69b2a90da\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.328428 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.328405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/585f71b9-5274-410c-9023-d7d69b2a90da-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-52rcf\" (UID: \"585f71b9-5274-410c-9023-d7d69b2a90da\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.341572 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.341547 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9b5\" (UniqueName: \"kubernetes.io/projected/585f71b9-5274-410c-9023-d7d69b2a90da-kube-api-access-7x9b5\") pod \"cert-manager-operator-controller-manager-55c444bf86-52rcf\" (UID: \"585f71b9-5274-410c-9023-d7d69b2a90da\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.408851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.408815 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" Jul 06 13:43:46.567313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.567282 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf"] Jul 06 13:43:46.571218 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:43:46.571189 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585f71b9_5274_410c_9023_d7d69b2a90da.slice/crio-4994cc62d941076ca2adfcf8026bfc13da7348da7be843b55ecccaaf88a72c7d WatchSource:0}: Error finding container 4994cc62d941076ca2adfcf8026bfc13da7348da7be843b55ecccaaf88a72c7d: Status 404 returned error can't find the container with id 4994cc62d941076ca2adfcf8026bfc13da7348da7be843b55ecccaaf88a72c7d Jul 06 13:43:46.877490 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:46.877457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" event={"ID":"585f71b9-5274-410c-9023-d7d69b2a90da","Type":"ContainerStarted","Data":"4994cc62d941076ca2adfcf8026bfc13da7348da7be843b55ecccaaf88a72c7d"} Jul 06 13:43:47.657374 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:47.657339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:47.758681 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:47.758627 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:49.893338 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:49.893299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" event={"ID":"585f71b9-5274-410c-9023-d7d69b2a90da","Type":"ContainerStarted","Data":"0a9064d56400c8b10f150d9e7106212e9c7db6e6a2c1fa434fc55fa2a38a12bb"} Jul 06 13:43:49.893745 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:49.893549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:49.913254 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:49.913192 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-52rcf" podStartSLOduration=1.475593282 podStartE2EDuration="3.913173321s" totalImagesPullingTime="2.437580039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:43:49.91123474 +0000 UTC m=+245.661048305" watchObservedRunningTime="2026-07-06 13:43:49.913173321 +0000 UTC m=+245.662986888" Jul 06 13:43:49.991016 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:49.990966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:50.001440 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.001409 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h4pbv"] Jul 06 13:43:50.003700 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.003682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.006332 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.006313 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 13:43:50.006438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.006321 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lmvvf\"" Jul 06 13:43:50.006438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.006413 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 13:43:50.017402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.017376 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h4pbv"] Jul 06 13:43:50.164261 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.164168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48fca70d-6859-46ba-b0db-e50e06da432e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h4pbv\" (UID: \"48fca70d-6859-46ba-b0db-e50e06da432e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.164261 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.164231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjmn\" (UniqueName: \"kubernetes.io/projected/48fca70d-6859-46ba-b0db-e50e06da432e-kube-api-access-hdjmn\") pod \"cert-manager-webhook-6486ffd997-h4pbv\" (UID: \"48fca70d-6859-46ba-b0db-e50e06da432e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.265645 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.265610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjmn\" (UniqueName: \"kubernetes.io/projected/48fca70d-6859-46ba-b0db-e50e06da432e-kube-api-access-hdjmn\") pod \"cert-manager-webhook-6486ffd997-h4pbv\" (UID: \"48fca70d-6859-46ba-b0db-e50e06da432e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.265852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.265675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48fca70d-6859-46ba-b0db-e50e06da432e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h4pbv\" (UID: \"48fca70d-6859-46ba-b0db-e50e06da432e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.288163 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.288132 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48fca70d-6859-46ba-b0db-e50e06da432e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h4pbv\" (UID: \"48fca70d-6859-46ba-b0db-e50e06da432e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.288360 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.288337 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjmn\" (UniqueName: \"kubernetes.io/projected/48fca70d-6859-46ba-b0db-e50e06da432e-kube-api-access-hdjmn\") pod \"cert-manager-webhook-6486ffd997-h4pbv\" (UID: \"48fca70d-6859-46ba-b0db-e50e06da432e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.313008 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.312976 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:50.322894 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.322602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:50.446910 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.446839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:50.479329 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.479305 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h4pbv"] Jul 06 13:43:50.480282 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:43:50.480259 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48fca70d_6859_46ba_b0db_e50e06da432e.slice/crio-da7ea84f0fc703b7738c9f16e54fe00f1002c76cb1f5de9c3a01614803b8140e WatchSource:0}: Error finding container da7ea84f0fc703b7738c9f16e54fe00f1002c76cb1f5de9c3a01614803b8140e: Status 404 returned error can't find the container with id da7ea84f0fc703b7738c9f16e54fe00f1002c76cb1f5de9c3a01614803b8140e Jul 06 13:43:50.609169 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.609130 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp"] Jul 06 13:43:50.612133 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.612117 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.614889 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.614862 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 13:43:50.615006 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.614977 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4td2\"" Jul 06 13:43:50.615006 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.614977 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 13:43:50.632188 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.632159 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp"] Jul 06 13:43:50.769277 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.769178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvtr\" (UniqueName: \"kubernetes.io/projected/465af7ed-c5cc-43e2-add6-9af0a19ca450-kube-api-access-kcvtr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.769277 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.769225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.769463 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.769314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.870374 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.870335 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.870545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.870397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvtr\" (UniqueName: \"kubernetes.io/projected/465af7ed-c5cc-43e2-add6-9af0a19ca450-kube-api-access-kcvtr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.870545 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.870439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.870765 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.870743 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.870823 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.870807 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.880037 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.880012 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvtr\" (UniqueName: \"kubernetes.io/projected/465af7ed-c5cc-43e2-add6-9af0a19ca450-kube-api-access-kcvtr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:50.898682 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.898644 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" event={"ID":"48fca70d-6859-46ba-b0db-e50e06da432e","Type":"ContainerStarted","Data":"da7ea84f0fc703b7738c9f16e54fe00f1002c76cb1f5de9c3a01614803b8140e"} Jul 06 13:43:50.922078 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:50.922047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:51.055607 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:51.055582 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp"] Jul 06 13:43:51.058067 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:43:51.058043 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465af7ed_c5cc_43e2_add6_9af0a19ca450.slice/crio-88b77fe8a314308be1bbaa0b9330e98b4775922b0425e55751b20f8b46f5f63c WatchSource:0}: Error finding container 88b77fe8a314308be1bbaa0b9330e98b4775922b0425e55751b20f8b46f5f63c: Status 404 returned error can't find the container with id 88b77fe8a314308be1bbaa0b9330e98b4775922b0425e55751b20f8b46f5f63c Jul 06 13:43:51.904369 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:51.904320 2562 generic.go:356] "Generic (PLEG): container finished" podID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerID="be80eb464ecc292976e7eadeecb20c696ceac313e18a01b5a1689a5e716b2792" exitCode=0 Jul 06 13:43:51.904895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:51.904400 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" event={"ID":"465af7ed-c5cc-43e2-add6-9af0a19ca450","Type":"ContainerDied","Data":"be80eb464ecc292976e7eadeecb20c696ceac313e18a01b5a1689a5e716b2792"} Jul 06 13:43:51.904895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:51.904457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" event={"ID":"465af7ed-c5cc-43e2-add6-9af0a19ca450","Type":"ContainerStarted","Data":"88b77fe8a314308be1bbaa0b9330e98b4775922b0425e55751b20f8b46f5f63c"} Jul 06 13:43:52.111869 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:52.111839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:52.805147 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:52.805109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:52.986779 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:52.986738 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx"] Jul 06 13:43:52.989556 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:52.989530 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:52.994405 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:52.994385 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2l8hp\"" Jul 06 13:43:53.014341 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.014313 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx"] Jul 06 13:43:53.090852 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.090749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba67cc99-3621-40c1-9d03-5f7115257382-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rlfhx\" (UID: \"ba67cc99-3621-40c1-9d03-5f7115257382\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.091021 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.090851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjjq\" (UniqueName: \"kubernetes.io/projected/ba67cc99-3621-40c1-9d03-5f7115257382-kube-api-access-9zjjq\") pod \"cert-manager-cainjector-559cdd8c87-rlfhx\" (UID: \"ba67cc99-3621-40c1-9d03-5f7115257382\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.192281 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.192237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjjq\" (UniqueName: \"kubernetes.io/projected/ba67cc99-3621-40c1-9d03-5f7115257382-kube-api-access-9zjjq\") pod \"cert-manager-cainjector-559cdd8c87-rlfhx\" (UID: \"ba67cc99-3621-40c1-9d03-5f7115257382\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.192462 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.192342 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba67cc99-3621-40c1-9d03-5f7115257382-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rlfhx\" (UID: \"ba67cc99-3621-40c1-9d03-5f7115257382\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.208039 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.208003 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjjq\" (UniqueName: \"kubernetes.io/projected/ba67cc99-3621-40c1-9d03-5f7115257382-kube-api-access-9zjjq\") pod \"cert-manager-cainjector-559cdd8c87-rlfhx\" (UID: \"ba67cc99-3621-40c1-9d03-5f7115257382\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.208305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.208284 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba67cc99-3621-40c1-9d03-5f7115257382-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rlfhx\" (UID: \"ba67cc99-3621-40c1-9d03-5f7115257382\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.301972 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.301937 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" Jul 06 13:43:53.467107 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.467051 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx"] Jul 06 13:43:53.470242 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:43:53.470203 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba67cc99_3621_40c1_9d03_5f7115257382.slice/crio-ea65ff73a5d1727e5d58bc656de30bbac30e782aba83d6f48209ee8c91bee018 WatchSource:0}: Error finding container ea65ff73a5d1727e5d58bc656de30bbac30e782aba83d6f48209ee8c91bee018: Status 404 returned error can't find the container with id ea65ff73a5d1727e5d58bc656de30bbac30e782aba83d6f48209ee8c91bee018 Jul 06 13:43:53.915514 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.915467 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" event={"ID":"48fca70d-6859-46ba-b0db-e50e06da432e","Type":"ContainerStarted","Data":"61d4b288ba9f33173b229f4f6b3b7413d74674216542530e3701a00a9ae5e5b3"} Jul 06 13:43:53.915718 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.915577 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:43:53.917182 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.917154 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" event={"ID":"ba67cc99-3621-40c1-9d03-5f7115257382","Type":"ContainerStarted","Data":"9120183d217dbf87050984f23e87f5e4da4cffd9b38f1d4c25c565f6e65520fe"} Jul 06 13:43:53.917285 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.917187 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" event={"ID":"ba67cc99-3621-40c1-9d03-5f7115257382","Type":"ContainerStarted","Data":"ea65ff73a5d1727e5d58bc656de30bbac30e782aba83d6f48209ee8c91bee018"} Jul 06 13:43:53.952704 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.952637 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" podStartSLOduration=2.111164492 podStartE2EDuration="4.952613721s" totalImagesPullingTime="2.841449229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:43:53.951468158 +0000 UTC m=+249.701281723" watchObservedRunningTime="2026-07-06 13:43:53.952613721 +0000 UTC m=+249.702427288" Jul 06 13:43:53.985413 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:53.985355 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rlfhx" podStartSLOduration=1.985333769 podStartE2EDuration="1.985333769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:43:53.98324443 +0000 UTC m=+249.733057996" watchObservedRunningTime="2026-07-06 13:43:53.985333769 +0000 UTC m=+249.735147333" Jul 06 13:43:55.310780 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:55.310693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:55.389162 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:55.389130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:43:55.928572 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:55.928540 2562 generic.go:356] "Generic (PLEG): container finished" podID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerID="50bc54d725ab5f3bdbecf2f37619fcf96a2c49d3c0ab82e8f3217dcd03996f12" exitCode=0 Jul 06 13:43:55.928770 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:55.928603 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" event={"ID":"465af7ed-c5cc-43e2-add6-9af0a19ca450","Type":"ContainerDied","Data":"50bc54d725ab5f3bdbecf2f37619fcf96a2c49d3c0ab82e8f3217dcd03996f12"} Jul 06 13:43:56.934565 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:56.934528 2562 generic.go:356] "Generic (PLEG): container finished" podID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerID="231e4f57059c3e152c1d874968ec57319bbccdb123547696924e5a18b76f346b" exitCode=0 Jul 06 13:43:56.934940 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:56.934618 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" event={"ID":"465af7ed-c5cc-43e2-add6-9af0a19ca450","Type":"ContainerDied","Data":"231e4f57059c3e152c1d874968ec57319bbccdb123547696924e5a18b76f346b"} Jul 06 13:43:58.068146 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.068115 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:58.137636 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.137604 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-util\") pod \"465af7ed-c5cc-43e2-add6-9af0a19ca450\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " Jul 06 13:43:58.137833 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.137680 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcvtr\" (UniqueName: \"kubernetes.io/projected/465af7ed-c5cc-43e2-add6-9af0a19ca450-kube-api-access-kcvtr\") pod \"465af7ed-c5cc-43e2-add6-9af0a19ca450\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " Jul 06 13:43:58.137833 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.137728 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-bundle\") pod \"465af7ed-c5cc-43e2-add6-9af0a19ca450\" (UID: \"465af7ed-c5cc-43e2-add6-9af0a19ca450\") " Jul 06 13:43:58.138142 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.138093 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-bundle" (OuterVolumeSpecName: "bundle") pod "465af7ed-c5cc-43e2-add6-9af0a19ca450" (UID: "465af7ed-c5cc-43e2-add6-9af0a19ca450"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:43:58.139734 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.139715 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465af7ed-c5cc-43e2-add6-9af0a19ca450-kube-api-access-kcvtr" (OuterVolumeSpecName: "kube-api-access-kcvtr") pod "465af7ed-c5cc-43e2-add6-9af0a19ca450" (UID: "465af7ed-c5cc-43e2-add6-9af0a19ca450"). InnerVolumeSpecName "kube-api-access-kcvtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:43:58.142317 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.142279 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-util" (OuterVolumeSpecName: "util") pod "465af7ed-c5cc-43e2-add6-9af0a19ca450" (UID: "465af7ed-c5cc-43e2-add6-9af0a19ca450"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:43:58.238495 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.238415 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcvtr\" (UniqueName: \"kubernetes.io/projected/465af7ed-c5cc-43e2-add6-9af0a19ca450-kube-api-access-kcvtr\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:58.238495 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.238440 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:58.238495 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.238450 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/465af7ed-c5cc-43e2-add6-9af0a19ca450-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:43:58.945042 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.945011 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" Jul 06 13:43:58.945042 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.945030 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtqnp" event={"ID":"465af7ed-c5cc-43e2-add6-9af0a19ca450","Type":"ContainerDied","Data":"88b77fe8a314308be1bbaa0b9330e98b4775922b0425e55751b20f8b46f5f63c"} Jul 06 13:43:58.945295 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:58.945074 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b77fe8a314308be1bbaa0b9330e98b4775922b0425e55751b20f8b46f5f63c" Jul 06 13:43:59.924578 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:43:59.924543 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h4pbv" Jul 06 13:44:00.717975 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:00.717945 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:02.552192 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:02.552155 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:02.682117 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:02.682062 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:03.557373 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.557334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:03.611693 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.611658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:03.917467 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917388 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8"] Jul 06 13:44:03.917744 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917732 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="util" Jul 06 13:44:03.917816 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917746 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="util" Jul 06 13:44:03.917816 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917755 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="extract" Jul 06 13:44:03.917816 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917760 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="extract" Jul 06 13:44:03.917816 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917776 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="pull" Jul 06 13:44:03.917981 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917781 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="pull" Jul 06 13:44:03.917981 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.917900 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="465af7ed-c5cc-43e2-add6-9af0a19ca450" containerName="extract" Jul 06 13:44:03.920432 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.920417 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:03.923135 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.923113 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 13:44:03.924347 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.924325 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 13:44:03.924427 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.924331 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-92mbg\"" Jul 06 13:44:03.929572 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.929541 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8"] Jul 06 13:44:03.977708 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.977672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbt5c\" (UniqueName: \"kubernetes.io/projected/21b6b12a-cbcb-41d4-83e9-c50ce708c331-kube-api-access-mbt5c\") pod \"openshift-lws-operator-fd8ccff4c-946b8\" (UID: \"21b6b12a-cbcb-41d4-83e9-c50ce708c331\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:03.977874 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:03.977740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21b6b12a-cbcb-41d4-83e9-c50ce708c331-tmp\") pod \"openshift-lws-operator-fd8ccff4c-946b8\" (UID: \"21b6b12a-cbcb-41d4-83e9-c50ce708c331\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:04.078727 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.078684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21b6b12a-cbcb-41d4-83e9-c50ce708c331-tmp\") pod \"openshift-lws-operator-fd8ccff4c-946b8\" (UID: \"21b6b12a-cbcb-41d4-83e9-c50ce708c331\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:04.078927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.078739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbt5c\" (UniqueName: \"kubernetes.io/projected/21b6b12a-cbcb-41d4-83e9-c50ce708c331-kube-api-access-mbt5c\") pod \"openshift-lws-operator-fd8ccff4c-946b8\" (UID: \"21b6b12a-cbcb-41d4-83e9-c50ce708c331\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:04.079097 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.079077 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21b6b12a-cbcb-41d4-83e9-c50ce708c331-tmp\") pod \"openshift-lws-operator-fd8ccff4c-946b8\" (UID: \"21b6b12a-cbcb-41d4-83e9-c50ce708c331\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:04.093573 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.093550 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbt5c\" (UniqueName: \"kubernetes.io/projected/21b6b12a-cbcb-41d4-83e9-c50ce708c331-kube-api-access-mbt5c\") pod \"openshift-lws-operator-fd8ccff4c-946b8\" (UID: \"21b6b12a-cbcb-41d4-83e9-c50ce708c331\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:04.230426 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.230330 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" Jul 06 13:44:04.362663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.362635 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8"] Jul 06 13:44:04.365295 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:44:04.365264 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b6b12a_cbcb_41d4_83e9_c50ce708c331.slice/crio-c803affba3e1d82efd86f1b355ca2036eb350a63e3fa296ebf23d193b5e21682 WatchSource:0}: Error finding container c803affba3e1d82efd86f1b355ca2036eb350a63e3fa296ebf23d193b5e21682: Status 404 returned error can't find the container with id c803affba3e1d82efd86f1b355ca2036eb350a63e3fa296ebf23d193b5e21682 Jul 06 13:44:04.973278 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:04.973247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" event={"ID":"21b6b12a-cbcb-41d4-83e9-c50ce708c331","Type":"ContainerStarted","Data":"c803affba3e1d82efd86f1b355ca2036eb350a63e3fa296ebf23d193b5e21682"} Jul 06 13:44:06.984892 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:06.984779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" event={"ID":"21b6b12a-cbcb-41d4-83e9-c50ce708c331","Type":"ContainerStarted","Data":"a68c2eb5fa52e66a6b5854a43760757edde4b05fc83e842acb80ffafb8e14501"} Jul 06 13:44:07.008340 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:07.008291 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-946b8" podStartSLOduration=1.7831075250000001 podStartE2EDuration="4.008274322s" totalImagesPullingTime="2.225166797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:44:07.00644878 +0000 UTC m=+262.756262345" watchObservedRunningTime="2026-07-06 13:44:07.008274322 +0000 UTC m=+262.758087887" Jul 06 13:44:07.745301 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:07.745264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:07.846013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:07.845981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:09.772203 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.772171 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm"] Jul 06 13:44:09.777509 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.777486 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.782310 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.782279 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 13:44:09.782449 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.782430 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 13:44:09.783585 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.783553 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q4td2\"" Jul 06 13:44:09.788859 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.788837 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm"] Jul 06 13:44:09.826937 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.826904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.826937 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.826942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jgx\" (UniqueName: \"kubernetes.io/projected/17a40904-1544-4c9e-8814-3860bc097ccb-kube-api-access-h9jgx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.827166 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.827039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.869684 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.869643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:09.927746 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.927713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.927746 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.927751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jgx\" (UniqueName: \"kubernetes.io/projected/17a40904-1544-4c9e-8814-3860bc097ccb-kube-api-access-h9jgx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.927977 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.927847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.928093 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.928074 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.928174 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.928157 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:09.937215 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:09.937194 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jgx\" (UniqueName: \"kubernetes.io/projected/17a40904-1544-4c9e-8814-3860bc097ccb-kube-api-access-h9jgx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:10.086961 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:10.086930 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:10.243990 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:10.243964 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm"] Jul 06 13:44:10.245753 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:44:10.245726 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a40904_1544_4c9e_8814_3860bc097ccb.slice/crio-a4cf7c64a13cfcbd565b8d8f765b32d3d577834fc8048630426f9ddc6db1bc08 WatchSource:0}: Error finding container a4cf7c64a13cfcbd565b8d8f765b32d3d577834fc8048630426f9ddc6db1bc08: Status 404 returned error can't find the container with id a4cf7c64a13cfcbd565b8d8f765b32d3d577834fc8048630426f9ddc6db1bc08 Jul 06 13:44:11.005014 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:11.004977 2562 generic.go:356] "Generic (PLEG): container finished" podID="17a40904-1544-4c9e-8814-3860bc097ccb" containerID="efa5c53d01c29c750332b30061cdd5231f65a09ecd46eb92faab298ebc2bbe5b" exitCode=0 Jul 06 13:44:11.005411 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:11.005056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" event={"ID":"17a40904-1544-4c9e-8814-3860bc097ccb","Type":"ContainerDied","Data":"efa5c53d01c29c750332b30061cdd5231f65a09ecd46eb92faab298ebc2bbe5b"} Jul 06 13:44:11.005411 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:11.005105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" event={"ID":"17a40904-1544-4c9e-8814-3860bc097ccb","Type":"ContainerStarted","Data":"a4cf7c64a13cfcbd565b8d8f765b32d3d577834fc8048630426f9ddc6db1bc08"} Jul 06 13:44:11.375758 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:11.375729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:11.433958 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:11.433884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:12.012046 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:12.011946 2562 generic.go:356] "Generic (PLEG): container finished" podID="17a40904-1544-4c9e-8814-3860bc097ccb" containerID="f12d8fcba337f8fbe1eee46b65f1e0c93e3980094de565ef3d09dc0a5b4333c3" exitCode=0 Jul 06 13:44:12.012046 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:12.012032 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" event={"ID":"17a40904-1544-4c9e-8814-3860bc097ccb","Type":"ContainerDied","Data":"f12d8fcba337f8fbe1eee46b65f1e0c93e3980094de565ef3d09dc0a5b4333c3"} Jul 06 13:44:13.018906 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:13.018871 2562 generic.go:356] "Generic (PLEG): container finished" podID="17a40904-1544-4c9e-8814-3860bc097ccb" containerID="f32ac0f782ae1b4080cf40c14abd6de8cc6caa52f609faf4ac4aaa76e2fee877" exitCode=0 Jul 06 13:44:13.019281 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:13.018952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" event={"ID":"17a40904-1544-4c9e-8814-3860bc097ccb","Type":"ContainerDied","Data":"f32ac0f782ae1b4080cf40c14abd6de8cc6caa52f609faf4ac4aaa76e2fee877"} Jul 06 13:44:14.161324 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.161298 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:14.265287 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.265255 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-util\") pod \"17a40904-1544-4c9e-8814-3860bc097ccb\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " Jul 06 13:44:14.265469 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.265300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9jgx\" (UniqueName: \"kubernetes.io/projected/17a40904-1544-4c9e-8814-3860bc097ccb-kube-api-access-h9jgx\") pod \"17a40904-1544-4c9e-8814-3860bc097ccb\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " Jul 06 13:44:14.265469 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.265379 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-bundle\") pod \"17a40904-1544-4c9e-8814-3860bc097ccb\" (UID: \"17a40904-1544-4c9e-8814-3860bc097ccb\") " Jul 06 13:44:14.266146 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.266110 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-bundle" (OuterVolumeSpecName: "bundle") pod "17a40904-1544-4c9e-8814-3860bc097ccb" (UID: "17a40904-1544-4c9e-8814-3860bc097ccb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:44:14.267440 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.267417 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a40904-1544-4c9e-8814-3860bc097ccb-kube-api-access-h9jgx" (OuterVolumeSpecName: "kube-api-access-h9jgx") pod "17a40904-1544-4c9e-8814-3860bc097ccb" (UID: "17a40904-1544-4c9e-8814-3860bc097ccb"). InnerVolumeSpecName "kube-api-access-h9jgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:44:14.270873 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.270846 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-util" (OuterVolumeSpecName: "util") pod "17a40904-1544-4c9e-8814-3860bc097ccb" (UID: "17a40904-1544-4c9e-8814-3860bc097ccb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:44:14.366000 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.365961 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:44:14.366000 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.365995 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9jgx\" (UniqueName: \"kubernetes.io/projected/17a40904-1544-4c9e-8814-3860bc097ccb-kube-api-access-h9jgx\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:44:14.366000 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:14.366005 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a40904-1544-4c9e-8814-3860bc097ccb-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:44:15.030020 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:15.029987 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" Jul 06 13:44:15.030020 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:15.030010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn47sm" event={"ID":"17a40904-1544-4c9e-8814-3860bc097ccb","Type":"ContainerDied","Data":"a4cf7c64a13cfcbd565b8d8f765b32d3d577834fc8048630426f9ddc6db1bc08"} Jul 06 13:44:15.030236 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:15.030056 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4cf7c64a13cfcbd565b8d8f765b32d3d577834fc8048630426f9ddc6db1bc08" Jul 06 13:44:24.752906 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:24.752870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:24.857027 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:24.856972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:25.535373 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:25.535333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:26.333578 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.333546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:26.474188 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.474156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:26.559000 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.558960 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd"] Jul 06 13:44:26.559375 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559342 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="pull" Jul 06 13:44:26.559375 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559364 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="pull" Jul 06 13:44:26.559532 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559394 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="extract" Jul 06 13:44:26.559532 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559401 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="extract" Jul 06 13:44:26.559532 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559414 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="util" Jul 06 13:44:26.559532 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559422 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="util" Jul 06 13:44:26.559532 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.559518 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a40904-1544-4c9e-8814-3860bc097ccb" containerName="extract" Jul 06 13:44:26.565901 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.565876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.572084 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.572060 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 13:44:26.572226 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.572150 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 13:44:26.596593 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.596511 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 13:44:26.597250 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.597229 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 13:44:26.601475 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.601459 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2q29q\"" Jul 06 13:44:26.670411 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.670374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06719a70-b17b-4407-bd61-b99e5fc36fce-tmp\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.670581 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.670419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06719a70-b17b-4407-bd61-b99e5fc36fce-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.670581 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.670512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06719a70-b17b-4407-bd61-b99e5fc36fce-manifests\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.670581 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.670541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/06719a70-b17b-4407-bd61-b99e5fc36fce-kube-api-access-d2wh2\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.670694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.670585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06719a70-b17b-4407-bd61-b99e5fc36fce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.693702 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.693669 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd"] Jul 06 13:44:26.771108 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.771076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06719a70-b17b-4407-bd61-b99e5fc36fce-manifests\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.771108 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.771115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/06719a70-b17b-4407-bd61-b99e5fc36fce-kube-api-access-d2wh2\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.771366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.771141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06719a70-b17b-4407-bd61-b99e5fc36fce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.771366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.771214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06719a70-b17b-4407-bd61-b99e5fc36fce-tmp\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.771366 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.771254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06719a70-b17b-4407-bd61-b99e5fc36fce-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.771590 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.771563 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06719a70-b17b-4407-bd61-b99e5fc36fce-manifests\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.773548 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.773524 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06719a70-b17b-4407-bd61-b99e5fc36fce-tmp\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.773659 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.773641 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06719a70-b17b-4407-bd61-b99e5fc36fce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.773898 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.773872 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06719a70-b17b-4407-bd61-b99e5fc36fce-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.780518 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.780470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:26.798239 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.798211 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/06719a70-b17b-4407-bd61-b99e5fc36fce-kube-api-access-d2wh2\") pod \"opendatahub-operator-controller-manager-6b559b5dcd-q99dd\" (UID: \"06719a70-b17b-4407-bd61-b99e5fc36fce\") " pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.876449 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.876367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:26.886951 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.886912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:26.999412 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:26.999379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:27.050345 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:27.050309 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd"] Jul 06 13:44:27.052631 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:44:27.052607 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06719a70_b17b_4407_bd61_b99e5fc36fce.slice/crio-fe05e2e3478190d630254f84d1d1a3f47c87f371164d33f82c418b6650d59514 WatchSource:0}: Error finding container fe05e2e3478190d630254f84d1d1a3f47c87f371164d33f82c418b6650d59514: Status 404 returned error can't find the container with id fe05e2e3478190d630254f84d1d1a3f47c87f371164d33f82c418b6650d59514 Jul 06 13:44:27.085564 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:27.085528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" event={"ID":"06719a70-b17b-4407-bd61-b99e5fc36fce","Type":"ContainerStarted","Data":"fe05e2e3478190d630254f84d1d1a3f47c87f371164d33f82c418b6650d59514"} Jul 06 13:44:27.096555 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:27.096527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:27.202469 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:27.202384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:27.334316 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:27.334279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:27.452007 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:27.451977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:30.104095 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:30.104055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" event={"ID":"06719a70-b17b-4407-bd61-b99e5fc36fce","Type":"ContainerStarted","Data":"e6a7894f43c4b5a4894d57a53f58d0d05f870580258f74ea718dcc205b8cc44a"} Jul 06 13:44:31.110199 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:31.110162 2562 generic.go:356] "Generic (PLEG): container finished" podID="06719a70-b17b-4407-bd61-b99e5fc36fce" containerID="e6a7894f43c4b5a4894d57a53f58d0d05f870580258f74ea718dcc205b8cc44a" exitCode=0 Jul 06 13:44:31.110642 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:31.110240 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" event={"ID":"06719a70-b17b-4407-bd61-b99e5fc36fce","Type":"ContainerDied","Data":"e6a7894f43c4b5a4894d57a53f58d0d05f870580258f74ea718dcc205b8cc44a"} Jul 06 13:44:32.117031 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:32.116993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" event={"ID":"06719a70-b17b-4407-bd61-b99e5fc36fce","Type":"ContainerStarted","Data":"fc22c18ebdba2c675fb6c4ae835b46170ebea31abf1cab37f8a030b6c257af82"} Jul 06 13:44:32.117453 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:32.117085 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:32.201838 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:32.201773 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" podStartSLOduration=1.8400113120000001 podStartE2EDuration="6.201757785s" totalImagesPullingTime="3.339208766s" totalInitContainerRuntime="1.022537707s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:44:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:44:32.201312611 +0000 UTC m=+287.951126200" watchObservedRunningTime="2026-07-06 13:44:32.201757785 +0000 UTC m=+287.951571349" Jul 06 13:44:40.133177 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:40.133144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:40.264090 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:40.264054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:40.393953 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:40.393871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:43.124562 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:43.124536 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b559b5dcd-q99dd" Jul 06 13:44:44.006972 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.006921 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh"] Jul 06 13:44:44.010068 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.010050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.023380 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.023355 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 13:44:44.023557 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.023410 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 13:44:44.024560 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.024534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 13:44:44.024663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.024654 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xzq6q\"" Jul 06 13:44:44.026624 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.026604 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh"] Jul 06 13:44:44.117562 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.117522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-cert\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.117740 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.117574 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8kt\" (UniqueName: \"kubernetes.io/projected/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-kube-api-access-pf8kt\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.117740 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.117631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-metrics-cert\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.117740 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.117657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-manager-config\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.218396 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.218358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-cert\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.218895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.218417 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8kt\" (UniqueName: \"kubernetes.io/projected/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-kube-api-access-pf8kt\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.218895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.218464 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-metrics-cert\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.218895 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.218584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-manager-config\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.219219 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.219201 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-manager-config\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.220821 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.220779 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-cert\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.220925 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.220851 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-metrics-cert\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.228802 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.228764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8kt\" (UniqueName: \"kubernetes.io/projected/2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf-kube-api-access-pf8kt\") pod \"lws-controller-manager-6ff5df8b5c-zr6mh\" (UID: \"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf\") " pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.318974 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.318942 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:44.462522 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.462496 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh"] Jul 06 13:44:44.464990 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:44:44.464957 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2bf2a8_2b4d_48b0_8cc6_329c4da05acf.slice/crio-c8f95203ef2a9a48f590457213b9a0c512f5d3ef200c12fcce7148d1f2980f57 WatchSource:0}: Error finding container c8f95203ef2a9a48f590457213b9a0c512f5d3ef200c12fcce7148d1f2980f57: Status 404 returned error can't find the container with id c8f95203ef2a9a48f590457213b9a0c512f5d3ef200c12fcce7148d1f2980f57 Jul 06 13:44:44.928144 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:44.928121 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 13:44:45.177960 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:45.177914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" event={"ID":"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf","Type":"ContainerStarted","Data":"c8f95203ef2a9a48f590457213b9a0c512f5d3ef200c12fcce7148d1f2980f57"} Jul 06 13:44:46.117303 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:46.117267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:46.185617 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:46.185580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" event={"ID":"2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf","Type":"ContainerStarted","Data":"ee786109bbc3060b75ad04935ee0612a970522439aa8ca7897ee4d944da1aae6"} Jul 06 13:44:46.185840 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:46.185726 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:46.291897 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:46.291853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:46.310651 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:46.310586 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" podStartSLOduration=1.784470029 podStartE2EDuration="3.3105655s" totalImagesPullingTime="1.526095471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:44:46.309316914 +0000 UTC m=+302.059130490" watchObservedRunningTime="2026-07-06 13:44:46.3105655 +0000 UTC m=+302.060379067" Jul 06 13:44:46.936962 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:46.936926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:47.259599 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.259477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:47.393781 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.393743 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx"] Jul 06 13:44:47.396519 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.396501 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.408842 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.408818 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 13:44:47.410015 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.409988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 13:44:47.410015 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.410009 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g6pvw\"" Jul 06 13:44:47.410194 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.410000 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 13:44:47.410194 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.410001 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 13:44:47.435760 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.435727 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx"] Jul 06 13:44:47.524878 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.524802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:47.554759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.554730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c03f7f14-040a-43f9-8372-27db7900efb4-tmp\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.554924 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.554899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f7f14-040a-43f9-8372-27db7900efb4-tls-certs\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.554972 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.554949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zc5r\" (UniqueName: \"kubernetes.io/projected/c03f7f14-040a-43f9-8372-27db7900efb4-kube-api-access-9zc5r\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.655907 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.655876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f7f14-040a-43f9-8372-27db7900efb4-tls-certs\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.656090 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.655938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc5r\" (UniqueName: \"kubernetes.io/projected/c03f7f14-040a-43f9-8372-27db7900efb4-kube-api-access-9zc5r\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.656090 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.655960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c03f7f14-040a-43f9-8372-27db7900efb4-tmp\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.658242 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.658217 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c03f7f14-040a-43f9-8372-27db7900efb4-tmp\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.658439 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.658419 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f7f14-040a-43f9-8372-27db7900efb4-tls-certs\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.679548 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.679510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:47.681057 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.681031 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc5r\" (UniqueName: \"kubernetes.io/projected/c03f7f14-040a-43f9-8372-27db7900efb4-kube-api-access-9zc5r\") pod \"kube-auth-proxy-85bfc8ccc9-pj6cx\" (UID: \"c03f7f14-040a-43f9-8372-27db7900efb4\") " pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.706856 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.706826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" Jul 06 13:44:47.887720 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.887657 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx"] Jul 06 13:44:47.892047 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:44:47.892009 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03f7f14_040a_43f9_8372_27db7900efb4.slice/crio-0ce1e7e25ebc1d585315d9f02a59ca0b6b84e264358539719599e360a8bb81f5 WatchSource:0}: Error finding container 0ce1e7e25ebc1d585315d9f02a59ca0b6b84e264358539719599e360a8bb81f5: Status 404 returned error can't find the container with id 0ce1e7e25ebc1d585315d9f02a59ca0b6b84e264358539719599e360a8bb81f5 Jul 06 13:44:47.894281 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.894265 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:44:47.895557 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:47.895530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:48.197916 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:48.197820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" event={"ID":"c03f7f14-040a-43f9-8372-27db7900efb4","Type":"ContainerStarted","Data":"0ce1e7e25ebc1d585315d9f02a59ca0b6b84e264358539719599e360a8bb81f5"} Jul 06 13:44:48.700698 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:48.700252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:48.802936 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:48.802847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:48.908361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:48.908291 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:49.496240 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:49.496209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:50.628900 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:50.628868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:51.216648 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:51.216613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" event={"ID":"c03f7f14-040a-43f9-8372-27db7900efb4","Type":"ContainerStarted","Data":"1631ceaa98edc6cfb7deab634a6188cc34adcf3354b8fc1e038831fe138a5390"} Jul 06 13:44:51.259123 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:51.259071 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85bfc8ccc9-pj6cx" podStartSLOduration=1.6837847350000001 podStartE2EDuration="4.259053015s" totalImagesPullingTime="2.57526828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:44:51.258545104 +0000 UTC m=+307.008358691" watchObservedRunningTime="2026-07-06 13:44:51.259053015 +0000 UTC m=+307.008866579" Jul 06 13:44:54.609527 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:54.609491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:54.706039 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:54.706005 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:55.648122 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:55.648091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:56.096438 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:56.096403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:56.256336 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:56.256301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:57.192897 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:57.192868 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6ff5df8b5c-zr6mh" Jul 06 13:44:59.013546 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.013495 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:59.118782 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.118747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:59.230265 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.230231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:59.329867 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.329834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:59.449203 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.449172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:59.541863 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.541816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:44:59.617587 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:44:59.617487 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:45:11.760370 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:45:11.760326 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:45:11.861071 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:45:11.861021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:45:30.580290 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:45:30.580252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:45:47.588047 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:45:47.587951 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:45:47.691638 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:45:47.691604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:45:47.962214 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:45:47.962121 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:03.178860 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.178821 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc"] Jul 06 13:46:03.181665 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.181647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.184640 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.184610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f8f7k\"" Jul 06 13:46:03.184640 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.184627 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 13:46:03.184858 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.184627 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 13:46:03.195241 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.195215 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc"] Jul 06 13:46:03.291475 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.291431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.291679 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.291514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.291679 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.291564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc6mj\" (UniqueName: \"kubernetes.io/projected/6fc842d6-81b4-4f45-801c-9a374e1252c2-kube-api-access-cc6mj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.391972 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.391934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.392169 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.391988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc6mj\" (UniqueName: \"kubernetes.io/projected/6fc842d6-81b4-4f45-801c-9a374e1252c2-kube-api-access-cc6mj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.392169 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.392014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.392320 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.392301 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.392376 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.392327 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.402488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.402456 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc6mj\" (UniqueName: \"kubernetes.io/projected/6fc842d6-81b4-4f45-801c-9a374e1252c2-kube-api-access-cc6mj\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.491466 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.491378 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:03.623320 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.623297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc"] Jul 06 13:46:03.625762 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:03.625735 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc842d6_81b4_4f45_801c_9a374e1252c2.slice/crio-503b568f910e7ed621847727781af128e4fb46420ece172c98dd3490b1ae5134 WatchSource:0}: Error finding container 503b568f910e7ed621847727781af128e4fb46420ece172c98dd3490b1ae5134: Status 404 returned error can't find the container with id 503b568f910e7ed621847727781af128e4fb46420ece172c98dd3490b1ae5134 Jul 06 13:46:03.760205 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.760127 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb"] Jul 06 13:46:03.762940 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.762921 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.771177 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.771154 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb"] Jul 06 13:46:03.795634 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.795608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.795742 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.795641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4gd\" (UniqueName: \"kubernetes.io/projected/e6052d41-3499-4aba-a87e-e7dcf7c62102-kube-api-access-fn4gd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.795742 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.795684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.896623 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.896590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.896623 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.896628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4gd\" (UniqueName: \"kubernetes.io/projected/e6052d41-3499-4aba-a87e-e7dcf7c62102-kube-api-access-fn4gd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.896872 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.896671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.897083 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.897059 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.897126 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.897080 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:03.906858 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:03.906835 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4gd\" (UniqueName: \"kubernetes.io/projected/e6052d41-3499-4aba-a87e-e7dcf7c62102-kube-api-access-fn4gd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:04.093598 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.093563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:04.160560 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.158962 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h"] Jul 06 13:46:04.163232 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.163149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.166352 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.166326 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h"] Jul 06 13:46:04.198806 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.198746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.199243 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.198858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.199243 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.198903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtrm\" (UniqueName: \"kubernetes.io/projected/8060ff05-57ac-42d7-b849-f81c671fb0b0-kube-api-access-nvtrm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.231615 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.231587 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb"] Jul 06 13:46:04.233199 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:04.233174 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6052d41_3499_4aba_a87e_e7dcf7c62102.slice/crio-d649c5b13d511ad68acaaf0d1c5438c16e58f66861ca18120c38c37ff92ade55 WatchSource:0}: Error finding container d649c5b13d511ad68acaaf0d1c5438c16e58f66861ca18120c38c37ff92ade55: Status 404 returned error can't find the container with id d649c5b13d511ad68acaaf0d1c5438c16e58f66861ca18120c38c37ff92ade55 Jul 06 13:46:04.300088 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.300055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.300253 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.300097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.300253 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.300123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtrm\" (UniqueName: \"kubernetes.io/projected/8060ff05-57ac-42d7-b849-f81c671fb0b0-kube-api-access-nvtrm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.300427 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.300407 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.300491 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.300446 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.308429 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.308403 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtrm\" (UniqueName: \"kubernetes.io/projected/8060ff05-57ac-42d7-b849-f81c671fb0b0-kube-api-access-nvtrm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.474188 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.474150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:04.556741 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.556710 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979"] Jul 06 13:46:04.559604 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.559582 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.569474 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.569445 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979"] Jul 06 13:46:04.571452 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.571419 2562 generic.go:356] "Generic (PLEG): container finished" podID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerID="7bcaf1db3fd2c39f3625cbab1f482424e2f41d63638e83a8970b2c81fbfb14ab" exitCode=0 Jul 06 13:46:04.571657 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.571625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" event={"ID":"6fc842d6-81b4-4f45-801c-9a374e1252c2","Type":"ContainerDied","Data":"7bcaf1db3fd2c39f3625cbab1f482424e2f41d63638e83a8970b2c81fbfb14ab"} Jul 06 13:46:04.571728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.571678 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" event={"ID":"6fc842d6-81b4-4f45-801c-9a374e1252c2","Type":"ContainerStarted","Data":"503b568f910e7ed621847727781af128e4fb46420ece172c98dd3490b1ae5134"} Jul 06 13:46:04.574704 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.574662 2562 generic.go:356] "Generic (PLEG): container finished" podID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerID="660d7b96b8d462f0a20229dbb987526af8b3ae47d0ecbac74430d618885f27f6" exitCode=0 Jul 06 13:46:04.574926 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.574717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" event={"ID":"e6052d41-3499-4aba-a87e-e7dcf7c62102","Type":"ContainerDied","Data":"660d7b96b8d462f0a20229dbb987526af8b3ae47d0ecbac74430d618885f27f6"} Jul 06 13:46:04.574926 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.574756 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" event={"ID":"e6052d41-3499-4aba-a87e-e7dcf7c62102","Type":"ContainerStarted","Data":"d649c5b13d511ad68acaaf0d1c5438c16e58f66861ca18120c38c37ff92ade55"} Jul 06 13:46:04.602865 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.602826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.603017 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.602919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/91542230-b669-4172-9839-de962f36d88f-kube-api-access-zn9j9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.603071 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.603051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.612011 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.611982 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h"] Jul 06 13:46:04.614798 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:04.614761 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8060ff05_57ac_42d7_b849_f81c671fb0b0.slice/crio-d222d28bd4d38e0491a588d0e19610aaa6ebb7fafdd66253d4d508bc749ad9d8 WatchSource:0}: Error finding container d222d28bd4d38e0491a588d0e19610aaa6ebb7fafdd66253d4d508bc749ad9d8: Status 404 returned error can't find the container with id d222d28bd4d38e0491a588d0e19610aaa6ebb7fafdd66253d4d508bc749ad9d8 Jul 06 13:46:04.704413 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.704374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/91542230-b669-4172-9839-de962f36d88f-kube-api-access-zn9j9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.704607 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.704428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.704607 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.704592 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.704856 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.704840 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.704963 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.704945 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.714833 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.714777 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/91542230-b669-4172-9839-de962f36d88f-kube-api-access-zn9j9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:04.873160 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:04.873118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:05.006184 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.006138 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979"] Jul 06 13:46:05.009436 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:05.009405 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91542230_b669_4172_9839_de962f36d88f.slice/crio-3f91f053791ab7411e861dc39403ddea12ee4fe431d972a5045b80a42912b716 WatchSource:0}: Error finding container 3f91f053791ab7411e861dc39403ddea12ee4fe431d972a5045b80a42912b716: Status 404 returned error can't find the container with id 3f91f053791ab7411e861dc39403ddea12ee4fe431d972a5045b80a42912b716 Jul 06 13:46:05.581080 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.581052 2562 generic.go:356] "Generic (PLEG): container finished" podID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerID="0808e00da628a7d0ab45f11e24566d3df607fd79cc1168827ac7217a20937c9b" exitCode=0 Jul 06 13:46:05.581474 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.581127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" event={"ID":"6fc842d6-81b4-4f45-801c-9a374e1252c2","Type":"ContainerDied","Data":"0808e00da628a7d0ab45f11e24566d3df607fd79cc1168827ac7217a20937c9b"} Jul 06 13:46:05.582873 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.582846 2562 generic.go:356] "Generic (PLEG): container finished" podID="91542230-b669-4172-9839-de962f36d88f" containerID="34aeec7c2055c483308a612582b6117c8ac60057ab342fa264a5e9729271ee34" exitCode=0 Jul 06 13:46:05.582966 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.582897 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" event={"ID":"91542230-b669-4172-9839-de962f36d88f","Type":"ContainerDied","Data":"34aeec7c2055c483308a612582b6117c8ac60057ab342fa264a5e9729271ee34"} Jul 06 13:46:05.582966 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.582950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" event={"ID":"91542230-b669-4172-9839-de962f36d88f","Type":"ContainerStarted","Data":"3f91f053791ab7411e861dc39403ddea12ee4fe431d972a5045b80a42912b716"} Jul 06 13:46:05.584775 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.584753 2562 generic.go:356] "Generic (PLEG): container finished" podID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerID="395418fe2078bac6532c4bbb8a4acf40f6724928508b5db33edceef13016677c" exitCode=0 Jul 06 13:46:05.584881 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.584820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" event={"ID":"e6052d41-3499-4aba-a87e-e7dcf7c62102","Type":"ContainerDied","Data":"395418fe2078bac6532c4bbb8a4acf40f6724928508b5db33edceef13016677c"} Jul 06 13:46:05.586553 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.586531 2562 generic.go:356] "Generic (PLEG): container finished" podID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerID="b5ff62b4f915d079145ddd27d6ead076c33104bf0dcad48fd1131407421608e2" exitCode=0 Jul 06 13:46:05.586660 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.586575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" event={"ID":"8060ff05-57ac-42d7-b849-f81c671fb0b0","Type":"ContainerDied","Data":"b5ff62b4f915d079145ddd27d6ead076c33104bf0dcad48fd1131407421608e2"} Jul 06 13:46:05.586660 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:05.586606 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" event={"ID":"8060ff05-57ac-42d7-b849-f81c671fb0b0","Type":"ContainerStarted","Data":"d222d28bd4d38e0491a588d0e19610aaa6ebb7fafdd66253d4d508bc749ad9d8"} Jul 06 13:46:06.596900 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.596867 2562 generic.go:356] "Generic (PLEG): container finished" podID="91542230-b669-4172-9839-de962f36d88f" containerID="799745e64939ba2f00af8ba42cb6049637174029ff70c9863f7578f2023ec6af" exitCode=0 Jul 06 13:46:06.597250 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.596902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" event={"ID":"91542230-b669-4172-9839-de962f36d88f","Type":"ContainerDied","Data":"799745e64939ba2f00af8ba42cb6049637174029ff70c9863f7578f2023ec6af"} Jul 06 13:46:06.599243 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.599216 2562 generic.go:356] "Generic (PLEG): container finished" podID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerID="6724e2e2ae72d9ca04ebd13945f6b7a1155b16e47bb7a26422698d1d15359f27" exitCode=0 Jul 06 13:46:06.599343 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.599295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" event={"ID":"e6052d41-3499-4aba-a87e-e7dcf7c62102","Type":"ContainerDied","Data":"6724e2e2ae72d9ca04ebd13945f6b7a1155b16e47bb7a26422698d1d15359f27"} Jul 06 13:46:06.600917 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.600895 2562 generic.go:356] "Generic (PLEG): container finished" podID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerID="a719767fe7d5ab6384c4e9b8ac53e9f20172df924278be0b8941667ad87709fc" exitCode=0 Jul 06 13:46:06.601015 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.600974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" event={"ID":"8060ff05-57ac-42d7-b849-f81c671fb0b0","Type":"ContainerDied","Data":"a719767fe7d5ab6384c4e9b8ac53e9f20172df924278be0b8941667ad87709fc"} Jul 06 13:46:06.603075 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.603057 2562 generic.go:356] "Generic (PLEG): container finished" podID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerID="bfc5bd702aedcaea383aa62162c35ab6db0a40ea4e1e888b625e5e9db107d461" exitCode=0 Jul 06 13:46:06.603148 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:06.603091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" event={"ID":"6fc842d6-81b4-4f45-801c-9a374e1252c2","Type":"ContainerDied","Data":"bfc5bd702aedcaea383aa62162c35ab6db0a40ea4e1e888b625e5e9db107d461"} Jul 06 13:46:07.609523 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.609494 2562 generic.go:356] "Generic (PLEG): container finished" podID="91542230-b669-4172-9839-de962f36d88f" containerID="f16b349cbcda953e47e79222a9a8e25c4f55162f47996e9f90272112f7e8042a" exitCode=0 Jul 06 13:46:07.609954 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.609589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" event={"ID":"91542230-b669-4172-9839-de962f36d88f","Type":"ContainerDied","Data":"f16b349cbcda953e47e79222a9a8e25c4f55162f47996e9f90272112f7e8042a"} Jul 06 13:46:07.611666 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.611639 2562 generic.go:356] "Generic (PLEG): container finished" podID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerID="c1c7f606233b399f164ac49d7a2c630d7549dcadf660a0bbdcd5731a567d913f" exitCode=0 Jul 06 13:46:07.611813 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.611704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" event={"ID":"8060ff05-57ac-42d7-b849-f81c671fb0b0","Type":"ContainerDied","Data":"c1c7f606233b399f164ac49d7a2c630d7549dcadf660a0bbdcd5731a567d913f"} Jul 06 13:46:07.768198 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.768173 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:07.770692 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.770674 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:07.831388 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-bundle\") pod \"6fc842d6-81b4-4f45-801c-9a374e1252c2\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " Jul 06 13:46:07.831388 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831391 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-bundle\") pod \"e6052d41-3499-4aba-a87e-e7dcf7c62102\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " Jul 06 13:46:07.831593 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-util\") pod \"6fc842d6-81b4-4f45-801c-9a374e1252c2\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " Jul 06 13:46:07.831593 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831432 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn4gd\" (UniqueName: \"kubernetes.io/projected/e6052d41-3499-4aba-a87e-e7dcf7c62102-kube-api-access-fn4gd\") pod \"e6052d41-3499-4aba-a87e-e7dcf7c62102\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " Jul 06 13:46:07.831593 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831460 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-util\") pod \"e6052d41-3499-4aba-a87e-e7dcf7c62102\" (UID: \"e6052d41-3499-4aba-a87e-e7dcf7c62102\") " Jul 06 13:46:07.831593 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831559 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc6mj\" (UniqueName: \"kubernetes.io/projected/6fc842d6-81b4-4f45-801c-9a374e1252c2-kube-api-access-cc6mj\") pod \"6fc842d6-81b4-4f45-801c-9a374e1252c2\" (UID: \"6fc842d6-81b4-4f45-801c-9a374e1252c2\") " Jul 06 13:46:07.832014 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.831981 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-bundle" (OuterVolumeSpecName: "bundle") pod "e6052d41-3499-4aba-a87e-e7dcf7c62102" (UID: "e6052d41-3499-4aba-a87e-e7dcf7c62102"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:07.832247 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.832220 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-bundle" (OuterVolumeSpecName: "bundle") pod "6fc842d6-81b4-4f45-801c-9a374e1252c2" (UID: "6fc842d6-81b4-4f45-801c-9a374e1252c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:07.834313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.834277 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc842d6-81b4-4f45-801c-9a374e1252c2-kube-api-access-cc6mj" (OuterVolumeSpecName: "kube-api-access-cc6mj") pod "6fc842d6-81b4-4f45-801c-9a374e1252c2" (UID: "6fc842d6-81b4-4f45-801c-9a374e1252c2"). InnerVolumeSpecName "kube-api-access-cc6mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:46:07.834313 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.834288 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6052d41-3499-4aba-a87e-e7dcf7c62102-kube-api-access-fn4gd" (OuterVolumeSpecName: "kube-api-access-fn4gd") pod "e6052d41-3499-4aba-a87e-e7dcf7c62102" (UID: "e6052d41-3499-4aba-a87e-e7dcf7c62102"). InnerVolumeSpecName "kube-api-access-fn4gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:46:07.836602 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.836577 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-util" (OuterVolumeSpecName: "util") pod "e6052d41-3499-4aba-a87e-e7dcf7c62102" (UID: "e6052d41-3499-4aba-a87e-e7dcf7c62102"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:07.837191 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.837172 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-util" (OuterVolumeSpecName: "util") pod "6fc842d6-81b4-4f45-801c-9a374e1252c2" (UID: "6fc842d6-81b4-4f45-801c-9a374e1252c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:07.932851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.932756 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:07.932851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.932801 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4gd\" (UniqueName: \"kubernetes.io/projected/e6052d41-3499-4aba-a87e-e7dcf7c62102-kube-api-access-fn4gd\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:07.932851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.932813 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:07.932851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.932822 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc6mj\" (UniqueName: \"kubernetes.io/projected/6fc842d6-81b4-4f45-801c-9a374e1252c2-kube-api-access-cc6mj\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:07.932851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.932830 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fc842d6-81b4-4f45-801c-9a374e1252c2-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:07.932851 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:07.932838 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6052d41-3499-4aba-a87e-e7dcf7c62102-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:08.618989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.618956 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" Jul 06 13:46:08.618989 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.618971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb" event={"ID":"e6052d41-3499-4aba-a87e-e7dcf7c62102","Type":"ContainerDied","Data":"d649c5b13d511ad68acaaf0d1c5438c16e58f66861ca18120c38c37ff92ade55"} Jul 06 13:46:08.619451 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.619018 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d649c5b13d511ad68acaaf0d1c5438c16e58f66861ca18120c38c37ff92ade55" Jul 06 13:46:08.620822 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.620775 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" Jul 06 13:46:08.620822 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.620808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc" event={"ID":"6fc842d6-81b4-4f45-801c-9a374e1252c2","Type":"ContainerDied","Data":"503b568f910e7ed621847727781af128e4fb46420ece172c98dd3490b1ae5134"} Jul 06 13:46:08.620982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.620846 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="503b568f910e7ed621847727781af128e4fb46420ece172c98dd3490b1ae5134" Jul 06 13:46:08.758736 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.758712 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:08.789232 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.789201 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:08.841009 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.840975 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvtrm\" (UniqueName: \"kubernetes.io/projected/8060ff05-57ac-42d7-b849-f81c671fb0b0-kube-api-access-nvtrm\") pod \"8060ff05-57ac-42d7-b849-f81c671fb0b0\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " Jul 06 13:46:08.841171 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841019 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/91542230-b669-4172-9839-de962f36d88f-kube-api-access-zn9j9\") pod \"91542230-b669-4172-9839-de962f36d88f\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " Jul 06 13:46:08.841171 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841053 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-util\") pod \"8060ff05-57ac-42d7-b849-f81c671fb0b0\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " Jul 06 13:46:08.841171 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841082 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-util\") pod \"91542230-b669-4172-9839-de962f36d88f\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " Jul 06 13:46:08.841171 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-bundle\") pod \"8060ff05-57ac-42d7-b849-f81c671fb0b0\" (UID: \"8060ff05-57ac-42d7-b849-f81c671fb0b0\") " Jul 06 13:46:08.841391 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841184 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-bundle\") pod \"91542230-b669-4172-9839-de962f36d88f\" (UID: \"91542230-b669-4172-9839-de962f36d88f\") " Jul 06 13:46:08.841754 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841674 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-bundle" (OuterVolumeSpecName: "bundle") pod "8060ff05-57ac-42d7-b849-f81c671fb0b0" (UID: "8060ff05-57ac-42d7-b849-f81c671fb0b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:08.841915 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.841890 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-bundle" (OuterVolumeSpecName: "bundle") pod "91542230-b669-4172-9839-de962f36d88f" (UID: "91542230-b669-4172-9839-de962f36d88f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:08.843248 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.843226 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91542230-b669-4172-9839-de962f36d88f-kube-api-access-zn9j9" (OuterVolumeSpecName: "kube-api-access-zn9j9") pod "91542230-b669-4172-9839-de962f36d88f" (UID: "91542230-b669-4172-9839-de962f36d88f"). InnerVolumeSpecName "kube-api-access-zn9j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:46:08.843334 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.843261 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8060ff05-57ac-42d7-b849-f81c671fb0b0-kube-api-access-nvtrm" (OuterVolumeSpecName: "kube-api-access-nvtrm") pod "8060ff05-57ac-42d7-b849-f81c671fb0b0" (UID: "8060ff05-57ac-42d7-b849-f81c671fb0b0"). InnerVolumeSpecName "kube-api-access-nvtrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:46:08.847291 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.847269 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-util" (OuterVolumeSpecName: "util") pod "8060ff05-57ac-42d7-b849-f81c671fb0b0" (UID: "8060ff05-57ac-42d7-b849-f81c671fb0b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:08.847826 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.847803 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-util" (OuterVolumeSpecName: "util") pod "91542230-b669-4172-9839-de962f36d88f" (UID: "91542230-b669-4172-9839-de962f36d88f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 13:46:08.941927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.941838 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtrm\" (UniqueName: \"kubernetes.io/projected/8060ff05-57ac-42d7-b849-f81c671fb0b0-kube-api-access-nvtrm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:08.941927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.941869 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/91542230-b669-4172-9839-de962f36d88f-kube-api-access-zn9j9\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:08.941927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.941879 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:08.941927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.941888 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-util\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:08.941927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.941896 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8060ff05-57ac-42d7-b849-f81c671fb0b0-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:08.941927 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:08.941904 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91542230-b669-4172-9839-de962f36d88f-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:09.627832 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:09.627777 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" Jul 06 13:46:09.627832 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:09.627820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979" event={"ID":"91542230-b669-4172-9839-de962f36d88f","Type":"ContainerDied","Data":"3f91f053791ab7411e861dc39403ddea12ee4fe431d972a5045b80a42912b716"} Jul 06 13:46:09.628338 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:09.627908 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f91f053791ab7411e861dc39403ddea12ee4fe431d972a5045b80a42912b716" Jul 06 13:46:09.629841 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:09.629779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" event={"ID":"8060ff05-57ac-42d7-b849-f81c671fb0b0","Type":"ContainerDied","Data":"d222d28bd4d38e0491a588d0e19610aaa6ebb7fafdd66253d4d508bc749ad9d8"} Jul 06 13:46:09.629841 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:09.629838 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h" Jul 06 13:46:09.630174 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:09.629841 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d222d28bd4d38e0491a588d0e19610aaa6ebb7fafdd66253d4d508bc749ad9d8" Jul 06 13:46:10.612567 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:10.612535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:14.205509 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:14.205470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:15.642600 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:15.642563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:16.028081 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:16.028000 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:17.132203 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:17.132168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:17.295054 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:17.295017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:18.626318 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:18.626289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:19.642518 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:19.642485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:21.089103 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:21.089074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:22.280641 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.280614 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:22.872699 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.872658 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8"] Jul 06 13:46:22.873055 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873042 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="util" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873057 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="util" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873067 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="util" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873073 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="util" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873079 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="pull" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873086 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="pull" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873093 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="pull" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873098 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="pull" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873106 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="util" Jul 06 13:46:22.873110 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873111 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="util" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873118 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873123 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873128 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873132 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873140 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873145 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873151 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="pull" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873156 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="pull" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873162 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873166 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873173 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="util" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873178 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="util" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873183 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="pull" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873187 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="pull" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873238 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc842d6-81b4-4f45-801c-9a374e1252c2" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873245 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6052d41-3499-4aba-a87e-e7dcf7c62102" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873252 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="91542230-b669-4172-9839-de962f36d88f" containerName="extract" Jul 06 13:46:22.873407 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.873261 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8060ff05-57ac-42d7-b849-f81c671fb0b0" containerName="extract" Jul 06 13:46:22.877759 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.877743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:22.882683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.882662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s76f5\"" Jul 06 13:46:22.885980 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.885963 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 13:46:22.886602 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.886585 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 13:46:22.886645 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.886621 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 13:46:22.952715 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.952676 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8"] Jul 06 13:46:22.969342 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:22.969314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrjp\" (UniqueName: \"kubernetes.io/projected/7ce4c2e8-953e-4446-9c7c-2a5015cf6273-kube-api-access-mmrjp\") pod \"dns-operator-controller-manager-864dd78c4c-wpcd8\" (UID: \"7ce4c2e8-953e-4446-9c7c-2a5015cf6273\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:23.070644 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.070607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrjp\" (UniqueName: \"kubernetes.io/projected/7ce4c2e8-953e-4446-9c7c-2a5015cf6273-kube-api-access-mmrjp\") pod \"dns-operator-controller-manager-864dd78c4c-wpcd8\" (UID: \"7ce4c2e8-953e-4446-9c7c-2a5015cf6273\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:23.085182 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.085152 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrjp\" (UniqueName: \"kubernetes.io/projected/7ce4c2e8-953e-4446-9c7c-2a5015cf6273-kube-api-access-mmrjp\") pod \"dns-operator-controller-manager-864dd78c4c-wpcd8\" (UID: \"7ce4c2e8-953e-4446-9c7c-2a5015cf6273\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:23.187694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.187604 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:23.359270 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.359235 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8"] Jul 06 13:46:23.372272 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:23.372243 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce4c2e8_953e_4446_9c7c_2a5015cf6273.slice/crio-9592d88f8d70b7fe81ae065aa4d57bec070791ef5892dad6a1777961100705d4 WatchSource:0}: Error finding container 9592d88f8d70b7fe81ae065aa4d57bec070791ef5892dad6a1777961100705d4: Status 404 returned error can't find the container with id 9592d88f8d70b7fe81ae065aa4d57bec070791ef5892dad6a1777961100705d4 Jul 06 13:46:23.445917 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.445829 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d476c456-ptdtq"] Jul 06 13:46:23.450456 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.450436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.475280 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.475248 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d476c456-ptdtq"] Jul 06 13:46:23.518476 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.518446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:23.575442 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-oauth-serving-cert\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.575600 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-serving-cert\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.575600 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-trusted-ca-bundle\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.575600 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575515 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5c69\" (UniqueName: \"kubernetes.io/projected/22613d14-3bf2-4c42-97bb-9dc38d8e28db-kube-api-access-f5c69\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.575723 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-config\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.575723 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575636 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-service-ca\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.575723 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.575669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-oauth-config\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.615499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.615466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:23.676283 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-oauth-serving-cert\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.676488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-serving-cert\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.676488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-trusted-ca-bundle\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.676488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5c69\" (UniqueName: \"kubernetes.io/projected/22613d14-3bf2-4c42-97bb-9dc38d8e28db-kube-api-access-f5c69\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.676488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-config\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.676488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-service-ca\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.676488 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.676457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-oauth-config\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.677233 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.677211 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-oauth-serving-cert\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.677398 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.677360 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-service-ca\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.677398 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.677382 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-trusted-ca-bundle\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.677801 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.677767 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-config\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.679009 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.678991 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-serving-cert\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.679077 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.679018 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22613d14-3bf2-4c42-97bb-9dc38d8e28db-console-oauth-config\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.687840 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.687815 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5c69\" (UniqueName: \"kubernetes.io/projected/22613d14-3bf2-4c42-97bb-9dc38d8e28db-kube-api-access-f5c69\") pod \"console-56d476c456-ptdtq\" (UID: \"22613d14-3bf2-4c42-97bb-9dc38d8e28db\") " pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.714355 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.714280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" event={"ID":"7ce4c2e8-953e-4446-9c7c-2a5015cf6273","Type":"ContainerStarted","Data":"9592d88f8d70b7fe81ae065aa4d57bec070791ef5892dad6a1777961100705d4"} Jul 06 13:46:23.760127 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.760067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:23.943596 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:23.943567 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d476c456-ptdtq"] Jul 06 13:46:23.945518 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:23.945491 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22613d14_3bf2_4c42_97bb_9dc38d8e28db.slice/crio-d76d6964168633f6d75a5bdcc34f14ad8225a8dd6dd57200ef6ce1869abf2f11 WatchSource:0}: Error finding container d76d6964168633f6d75a5bdcc34f14ad8225a8dd6dd57200ef6ce1869abf2f11: Status 404 returned error can't find the container with id d76d6964168633f6d75a5bdcc34f14ad8225a8dd6dd57200ef6ce1869abf2f11 Jul 06 13:46:24.722123 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:24.722089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d476c456-ptdtq" event={"ID":"22613d14-3bf2-4c42-97bb-9dc38d8e28db","Type":"ContainerStarted","Data":"d05fd8d7c6147130ef3efb912bd9d7cb827ed6ef52591771b9772b645257ad03"} Jul 06 13:46:24.722532 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:24.722139 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d476c456-ptdtq" event={"ID":"22613d14-3bf2-4c42-97bb-9dc38d8e28db","Type":"ContainerStarted","Data":"d76d6964168633f6d75a5bdcc34f14ad8225a8dd6dd57200ef6ce1869abf2f11"} Jul 06 13:46:24.742587 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:24.742535 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d476c456-ptdtq" podStartSLOduration=1.742515482 podStartE2EDuration="1.742515482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:46:24.742212114 +0000 UTC m=+400.492025671" watchObservedRunningTime="2026-07-06 13:46:24.742515482 +0000 UTC m=+400.492329025" Jul 06 13:46:24.840610 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:24.840572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:24.990772 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:24.990695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:25.628842 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.628798 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v"] Jul 06 13:46:25.629378 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.629351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:25.632773 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.632750 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.638206 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.638181 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f8f7k\"" Jul 06 13:46:25.638486 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.638471 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 13:46:25.638591 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.638521 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 13:46:25.651140 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.651110 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v"] Jul 06 13:46:25.693516 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.693483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndsj\" (UniqueName: \"kubernetes.io/projected/92681fff-cf12-42ad-b7bf-7ae7936e2691-kube-api-access-zndsj\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.693681 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.693570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/92681fff-cf12-42ad-b7bf-7ae7936e2691-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.693681 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.693611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92681fff-cf12-42ad-b7bf-7ae7936e2691-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.794499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.794352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/92681fff-cf12-42ad-b7bf-7ae7936e2691-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.794499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.794430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92681fff-cf12-42ad-b7bf-7ae7936e2691-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.795053 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.794630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndsj\" (UniqueName: \"kubernetes.io/projected/92681fff-cf12-42ad-b7bf-7ae7936e2691-kube-api-access-zndsj\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.795146 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.795123 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/92681fff-cf12-42ad-b7bf-7ae7936e2691-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:25.795210 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:46:25.795145 2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 13:46:25.795210 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:46:25.795202 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92681fff-cf12-42ad-b7bf-7ae7936e2691-plugin-serving-cert podName:92681fff-cf12-42ad-b7bf-7ae7936e2691 nodeName:}" failed. No retries permitted until 2026-07-06 13:46:26.295183834 +0000 UTC m=+402.044997409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/92681fff-cf12-42ad-b7bf-7ae7936e2691-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-hgw2v" (UID: "92681fff-cf12-42ad-b7bf-7ae7936e2691") : secret "plugin-serving-cert" not found Jul 06 13:46:25.832947 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:25.832916 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndsj\" (UniqueName: \"kubernetes.io/projected/92681fff-cf12-42ad-b7bf-7ae7936e2691-kube-api-access-zndsj\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:26.298646 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.298610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92681fff-cf12-42ad-b7bf-7ae7936e2691-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:26.300985 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.300964 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/92681fff-cf12-42ad-b7bf-7ae7936e2691-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hgw2v\" (UID: \"92681fff-cf12-42ad-b7bf-7ae7936e2691\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:26.319076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.319017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:26.543137 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.543099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" Jul 06 13:46:26.684432 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.684404 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v"] Jul 06 13:46:26.685966 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:46:26.685935 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92681fff_cf12_42ad_b7bf_7ae7936e2691.slice/crio-64a0917d4c580b8658a8a616b1a43fff0aaf47cc06489e150bf09456adb050ed WatchSource:0}: Error finding container 64a0917d4c580b8658a8a616b1a43fff0aaf47cc06489e150bf09456adb050ed: Status 404 returned error can't find the container with id 64a0917d4c580b8658a8a616b1a43fff0aaf47cc06489e150bf09456adb050ed Jul 06 13:46:26.733803 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.733752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" event={"ID":"92681fff-cf12-42ad-b7bf-7ae7936e2691","Type":"ContainerStarted","Data":"64a0917d4c580b8658a8a616b1a43fff0aaf47cc06489e150bf09456adb050ed"} Jul 06 13:46:26.735288 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.735266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" event={"ID":"7ce4c2e8-953e-4446-9c7c-2a5015cf6273","Type":"ContainerStarted","Data":"86ed3368e5e794b54e3a66484794aaa1b8d2f26ba1ebb22366164c1988b545df"} Jul 06 13:46:26.735451 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.735431 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:26.756663 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:26.756618 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" podStartSLOduration=2.435870089 podStartE2EDuration="4.756601837s" totalImagesPullingTime="2.320731748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:46:26.755025044 +0000 UTC m=+402.504838636" watchObservedRunningTime="2026-07-06 13:46:26.756601837 +0000 UTC m=+402.506415401" Jul 06 13:46:33.760865 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:33.760826 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:33.761323 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:33.760892 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:33.767517 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:33.767492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:33.783343 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:33.783317 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56d476c456-ptdtq" Jul 06 13:46:33.986129 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:33.986096 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64786c6f75-hxfh6"] Jul 06 13:46:34.118338 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:34.118309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:34.222100 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:34.222060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:34.356639 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:34.356601 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:34.444260 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:34.444165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:34.913321 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:34.913284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:35.027322 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:35.027268 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:37.743593 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:37.743558 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wpcd8" Jul 06 13:46:38.057129 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:38.057089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:38.233982 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:38.233932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:40.538728 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:40.538695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:40.687650 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:40.687601 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:46:48.872572 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:48.872476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" event={"ID":"92681fff-cf12-42ad-b7bf-7ae7936e2691","Type":"ContainerStarted","Data":"d4823c15dd9013363efdbc5b936d69e057a013ebe5606a8698ef2cd6382a2849"} Jul 06 13:46:48.901740 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:48.901678 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hgw2v" podStartSLOduration=1.9886379870000002 podStartE2EDuration="23.9016605s" totalImagesPullingTime="21.913022513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:46:48.899727782 +0000 UTC m=+424.649541348" watchObservedRunningTime="2026-07-06 13:46:48.9016605 +0000 UTC m=+424.651474062" Jul 06 13:46:59.013613 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.013566 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-64786c6f75-hxfh6" podUID="a2ac2931-157e-40e9-be54-36f444ecdfef" containerName="console" containerID="cri-o://1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863" gracePeriod=15 Jul 06 13:46:59.261903 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.261876 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:46:59.303278 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303245 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l28n\" (UniqueName: \"kubernetes.io/projected/a2ac2931-157e-40e9-be54-36f444ecdfef-kube-api-access-9l28n\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303471 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-serving-cert\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303471 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303335 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-service-ca\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303594 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-oauth-serving-cert\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303594 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303522 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-oauth-config\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303710 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303592 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-trusted-ca-bundle\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303710 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303617 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-console-config\") pod \"a2ac2931-157e-40e9-be54-36f444ecdfef\" (UID: \"a2ac2931-157e-40e9-be54-36f444ecdfef\") " Jul 06 13:46:59.303710 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303700 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-service-ca" (OuterVolumeSpecName: "service-ca") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:46:59.303985 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.303944 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-service-ca\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.304202 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.304173 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:46:59.304340 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.304237 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-console-config" (OuterVolumeSpecName: "console-config") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:46:59.304340 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.304233 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 13:46:59.305595 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.305569 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ac2931-157e-40e9-be54-36f444ecdfef-kube-api-access-9l28n" (OuterVolumeSpecName: "kube-api-access-9l28n") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "kube-api-access-9l28n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:46:59.305694 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.305677 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:46:59.306223 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.306194 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a2ac2931-157e-40e9-be54-36f444ecdfef" (UID: "a2ac2931-157e-40e9-be54-36f444ecdfef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 13:46:59.404499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.404456 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l28n\" (UniqueName: \"kubernetes.io/projected/a2ac2931-157e-40e9-be54-36f444ecdfef-kube-api-access-9l28n\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.404499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.404493 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.404499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.404504 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-oauth-serving-cert\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.404499 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.404512 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2ac2931-157e-40e9-be54-36f444ecdfef-console-oauth-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.404770 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.404520 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-trusted-ca-bundle\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.404770 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.404529 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2ac2931-157e-40e9-be54-36f444ecdfef-console-config\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:46:59.941317 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.941281 2562 generic.go:356] "Generic (PLEG): container finished" podID="a2ac2931-157e-40e9-be54-36f444ecdfef" containerID="1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863" exitCode=0 Jul 06 13:46:59.941517 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.941329 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64786c6f75-hxfh6" Jul 06 13:46:59.941517 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.941378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64786c6f75-hxfh6" event={"ID":"a2ac2931-157e-40e9-be54-36f444ecdfef","Type":"ContainerDied","Data":"1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863"} Jul 06 13:46:59.941517 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.941428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64786c6f75-hxfh6" event={"ID":"a2ac2931-157e-40e9-be54-36f444ecdfef","Type":"ContainerDied","Data":"c6a5fbce53dfdc1ebe07cc6d338f523b6aaa7178fff9b0d5db7896f67f93c0f8"} Jul 06 13:46:59.941517 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.941453 2562 scope.go:117] "RemoveContainer" containerID="1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863" Jul 06 13:46:59.950183 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.950169 2562 scope.go:117] "RemoveContainer" containerID="1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863" Jul 06 13:46:59.950436 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:46:59.950419 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863\": container with ID starting with 1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863 not found: ID does not exist" containerID="1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863" Jul 06 13:46:59.950482 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.950443 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863"} err="failed to get container status \"1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863\": rpc error: code = NotFound desc = could not find container \"1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863\": container with ID starting with 1ae4152768cc9eb1e3e3c3fc05fb0e23d1401725de4c0f012f191b6a02fe6863 not found: ID does not exist" Jul 06 13:46:59.975413 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.975381 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64786c6f75-hxfh6"] Jul 06 13:46:59.977633 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:46:59.977611 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64786c6f75-hxfh6"] Jul 06 13:47:01.543607 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:01.543579 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ac2931-157e-40e9-be54-36f444ecdfef" path="/var/lib/kubelet/pods/a2ac2931-157e-40e9-be54-36f444ecdfef/volumes" Jul 06 13:47:12.643328 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:12.643247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:12.956770 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:12.956690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:15.838448 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:15.838416 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:15.948482 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:15.948453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:16.075963 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.075924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:16.293306 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.293269 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:47:16.293753 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.293728 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ac2931-157e-40e9-be54-36f444ecdfef" containerName="console" Jul 06 13:47:16.293753 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.293748 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ac2931-157e-40e9-be54-36f444ecdfef" containerName="console" Jul 06 13:47:16.293920 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.293838 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ac2931-157e-40e9-be54-36f444ecdfef" containerName="console" Jul 06 13:47:16.350362 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.350324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:47:16.350523 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.350443 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.355186 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.355164 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 13:47:16.363473 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.363446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:47:16.440273 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.440239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4ceec97-7568-4765-ac03-c8a388c17950-config-file\") pod \"limitador-limitador-54d6d988d5-5wdlq\" (UID: \"a4ceec97-7568-4765-ac03-c8a388c17950\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.440466 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.440297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblz5\" (UniqueName: \"kubernetes.io/projected/a4ceec97-7568-4765-ac03-c8a388c17950-kube-api-access-xblz5\") pod \"limitador-limitador-54d6d988d5-5wdlq\" (UID: \"a4ceec97-7568-4765-ac03-c8a388c17950\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.540910 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.540878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblz5\" (UniqueName: \"kubernetes.io/projected/a4ceec97-7568-4765-ac03-c8a388c17950-kube-api-access-xblz5\") pod \"limitador-limitador-54d6d988d5-5wdlq\" (UID: \"a4ceec97-7568-4765-ac03-c8a388c17950\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.541098 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.540929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4ceec97-7568-4765-ac03-c8a388c17950-config-file\") pod \"limitador-limitador-54d6d988d5-5wdlq\" (UID: \"a4ceec97-7568-4765-ac03-c8a388c17950\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.541482 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.541463 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a4ceec97-7568-4765-ac03-c8a388c17950-config-file\") pod \"limitador-limitador-54d6d988d5-5wdlq\" (UID: \"a4ceec97-7568-4765-ac03-c8a388c17950\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.555122 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.555057 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblz5\" (UniqueName: \"kubernetes.io/projected/a4ceec97-7568-4765-ac03-c8a388c17950-kube-api-access-xblz5\") pod \"limitador-limitador-54d6d988d5-5wdlq\" (UID: \"a4ceec97-7568-4765-ac03-c8a388c17950\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.661576 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.661537 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:16.809361 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.809335 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:47:16.810951 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:47:16.810922 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ceec97_7568_4765_ac03_c8a388c17950.slice/crio-0087624c817c62f3c53edae1185fa9b575462cae5ba96ab6bb357c60cce53ec7 WatchSource:0}: Error finding container 0087624c817c62f3c53edae1185fa9b575462cae5ba96ab6bb357c60cce53ec7: Status 404 returned error can't find the container with id 0087624c817c62f3c53edae1185fa9b575462cae5ba96ab6bb357c60cce53ec7 Jul 06 13:47:16.831100 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.831073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:16.913578 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:16.913546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:17.031043 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:17.031008 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" event={"ID":"a4ceec97-7568-4765-ac03-c8a388c17950","Type":"ContainerStarted","Data":"0087624c817c62f3c53edae1185fa9b575462cae5ba96ab6bb357c60cce53ec7"} Jul 06 13:47:20.049955 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:20.049921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" event={"ID":"a4ceec97-7568-4765-ac03-c8a388c17950","Type":"ContainerStarted","Data":"03f1b5168072e3f5f7f5a7bf4b1e7826709578c3cd0ef2f05be57b0bb8a484ff"} Jul 06 13:47:20.050365 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:20.050047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:20.072013 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:20.071964 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" podStartSLOduration=1.558953277 podStartE2EDuration="4.071948333s" totalImagesPullingTime="2.512995056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:47:20.070465907 +0000 UTC m=+455.820279473" watchObservedRunningTime="2026-07-06 13:47:20.071948333 +0000 UTC m=+455.821761897" Jul 06 13:47:31.056684 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:31.056655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5wdlq" Jul 06 13:47:37.420527 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:37.420484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:50.073831 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.073750 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:50.542305 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.542248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:50.613676 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.613643 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-544967bbdd-gkt5s"] Jul 06 13:47:50.618423 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.618402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:50.627633 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.627613 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n7mbc\"" Jul 06 13:47:50.627894 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.627872 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vsp\" (UniqueName: \"kubernetes.io/projected/6e1ba7ce-b655-446a-9ced-a525f09bdad2-kube-api-access-t5vsp\") pod \"authorino-544967bbdd-gkt5s\" (UID: \"6e1ba7ce-b655-446a-9ced-a525f09bdad2\") " pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:50.638257 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.638234 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544967bbdd-gkt5s"] Jul 06 13:47:50.728381 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.728339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vsp\" (UniqueName: \"kubernetes.io/projected/6e1ba7ce-b655-446a-9ced-a525f09bdad2-kube-api-access-t5vsp\") pod \"authorino-544967bbdd-gkt5s\" (UID: \"6e1ba7ce-b655-446a-9ced-a525f09bdad2\") " pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:50.740529 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.740503 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vsp\" (UniqueName: \"kubernetes.io/projected/6e1ba7ce-b655-446a-9ced-a525f09bdad2-kube-api-access-t5vsp\") pod \"authorino-544967bbdd-gkt5s\" (UID: \"6e1ba7ce-b655-446a-9ced-a525f09bdad2\") " pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:50.927375 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.927283 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:50.996099 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:50.996072 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544967bbdd-gkt5s"] Jul 06 13:47:51.090049 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:47:51.090000 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e1ba7ce_b655_446a_9ced_a525f09bdad2.slice/crio-4afbdf3afda0dabc254bffac9726d4f40877a6ecf33643cf0779af14c040d609 WatchSource:0}: Error finding container 4afbdf3afda0dabc254bffac9726d4f40877a6ecf33643cf0779af14c040d609: Status 404 returned error can't find the container with id 4afbdf3afda0dabc254bffac9726d4f40877a6ecf33643cf0779af14c040d609 Jul 06 13:47:51.093988 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:51.093961 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544967bbdd-gkt5s"] Jul 06 13:47:51.215035 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:51.214955 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544967bbdd-gkt5s" event={"ID":"6e1ba7ce-b655-446a-9ced-a525f09bdad2","Type":"ContainerStarted","Data":"4afbdf3afda0dabc254bffac9726d4f40877a6ecf33643cf0779af14c040d609"} Jul 06 13:47:55.242167 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.242107 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544967bbdd-gkt5s" event={"ID":"6e1ba7ce-b655-446a-9ced-a525f09bdad2","Type":"ContainerStarted","Data":"a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7"} Jul 06 13:47:55.242167 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.242147 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-544967bbdd-gkt5s" podUID="6e1ba7ce-b655-446a-9ced-a525f09bdad2" containerName="authorino" containerID="cri-o://a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7" gracePeriod=30 Jul 06 13:47:55.264208 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.264162 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-544967bbdd-gkt5s" podStartSLOduration=1.810335187 podStartE2EDuration="5.264146366s" totalImagesPullingTime="3.453811179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:47:55.262859224 +0000 UTC m=+491.012672788" watchObservedRunningTime="2026-07-06 13:47:55.264146366 +0000 UTC m=+491.013959934" Jul 06 13:47:55.497173 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.497105 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:55.572298 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.572258 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5vsp\" (UniqueName: \"kubernetes.io/projected/6e1ba7ce-b655-446a-9ced-a525f09bdad2-kube-api-access-t5vsp\") pod \"6e1ba7ce-b655-446a-9ced-a525f09bdad2\" (UID: \"6e1ba7ce-b655-446a-9ced-a525f09bdad2\") " Jul 06 13:47:55.574288 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.574263 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1ba7ce-b655-446a-9ced-a525f09bdad2-kube-api-access-t5vsp" (OuterVolumeSpecName: "kube-api-access-t5vsp") pod "6e1ba7ce-b655-446a-9ced-a525f09bdad2" (UID: "6e1ba7ce-b655-446a-9ced-a525f09bdad2"). InnerVolumeSpecName "kube-api-access-t5vsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 13:47:55.673004 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:55.672965 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5vsp\" (UniqueName: \"kubernetes.io/projected/6e1ba7ce-b655-446a-9ced-a525f09bdad2-kube-api-access-t5vsp\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 13:47:56.248194 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.248159 2562 generic.go:356] "Generic (PLEG): container finished" podID="6e1ba7ce-b655-446a-9ced-a525f09bdad2" containerID="a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7" exitCode=0 Jul 06 13:47:56.248194 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.248196 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544967bbdd-gkt5s" Jul 06 13:47:56.248673 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.248236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544967bbdd-gkt5s" event={"ID":"6e1ba7ce-b655-446a-9ced-a525f09bdad2","Type":"ContainerDied","Data":"a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7"} Jul 06 13:47:56.248673 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.248278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544967bbdd-gkt5s" event={"ID":"6e1ba7ce-b655-446a-9ced-a525f09bdad2","Type":"ContainerDied","Data":"4afbdf3afda0dabc254bffac9726d4f40877a6ecf33643cf0779af14c040d609"} Jul 06 13:47:56.248673 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.248295 2562 scope.go:117] "RemoveContainer" containerID="a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7" Jul 06 13:47:56.257599 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.257581 2562 scope.go:117] "RemoveContainer" containerID="a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7" Jul 06 13:47:56.257843 ip-10-0-134-202 kubenswrapper[2562]: E0706 13:47:56.257822 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7\": container with ID starting with a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7 not found: ID does not exist" containerID="a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7" Jul 06 13:47:56.257898 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.257850 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7"} err="failed to get container status \"a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7\": rpc error: code = NotFound desc = could not find container \"a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7\": container with ID starting with a4a5c818ece0a99545444c4e6168eb209a648adcc334b3917ec2aab408425fd7 not found: ID does not exist" Jul 06 13:47:56.271590 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.271569 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544967bbdd-gkt5s"] Jul 06 13:47:56.275465 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:56.275446 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-544967bbdd-gkt5s"] Jul 06 13:47:57.315547 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:57.315512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:57.431810 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:57.431756 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:57.536286 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:57.536248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:47:57.544207 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:57.544178 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1ba7ce-b655-446a-9ced-a525f09bdad2" path="/var/lib/kubelet/pods/6e1ba7ce-b655-446a-9ced-a525f09bdad2/volumes" Jul 06 13:47:57.610164 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:47:57.610063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:00.280243 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:00.280214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:00.376372 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:00.376338 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:31.463919 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:31.463880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:31.856828 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:31.856748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:32.144334 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:32.144230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:32.648122 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:32.648081 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:48:32.846140 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:32.846080 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:32.952834 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:32.952693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:43.084192 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:43.084086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:43.182981 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:43.182947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:43.444731 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:43.444648 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:52.240708 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:52.240679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:58.440281 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:58.440243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:48:59.019590 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:48:59.019556 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:49:18.937035 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:18.937002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:19.608929 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:19.608888 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:49:23.566284 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:23.566246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:24.509061 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:24.509025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:49:29.427061 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:29.427026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:29.489738 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:29.489708 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:37.683925 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.683885 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh"] Jul 06 13:49:37.684311 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.684236 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1ba7ce-b655-446a-9ced-a525f09bdad2" containerName="authorino" Jul 06 13:49:37.684311 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.684247 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1ba7ce-b655-446a-9ced-a525f09bdad2" containerName="authorino" Jul 06 13:49:37.684311 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.684310 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1ba7ce-b655-446a-9ced-a525f09bdad2" containerName="authorino" Jul 06 13:49:37.687426 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.687409 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.693092 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.693066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 13:49:37.693380 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.693365 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 13:49:37.693463 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.693364 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xpd76\"" Jul 06 13:49:37.693911 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.693892 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 13:49:37.701601 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.701575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh"] Jul 06 13:49:37.723699 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.723666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:37.834184 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.834144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.834369 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.834207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.834369 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.834275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjtz\" (UniqueName: \"kubernetes.io/projected/60425df0-fb04-4d4b-92bf-ede71cdf72b9-kube-api-access-mrjtz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.834369 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.834324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60425df0-fb04-4d4b-92bf-ede71cdf72b9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.834369 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.834351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.834526 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.834378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935227 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935227 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjtz\" (UniqueName: \"kubernetes.io/projected/60425df0-fb04-4d4b-92bf-ede71cdf72b9-kube-api-access-mrjtz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60425df0-fb04-4d4b-92bf-ede71cdf72b9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935470 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935699 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935672 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935753 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935687 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.935753 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.935734 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.937441 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.937417 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60425df0-fb04-4d4b-92bf-ede71cdf72b9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.937722 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.937702 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60425df0-fb04-4d4b-92bf-ede71cdf72b9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.945194 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.945170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjtz\" (UniqueName: \"kubernetes.io/projected/60425df0-fb04-4d4b-92bf-ede71cdf72b9-kube-api-access-mrjtz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh\" (UID: \"60425df0-fb04-4d4b-92bf-ede71cdf72b9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:37.999149 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:37.999099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:38.140201 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:38.140174 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh"] Jul 06 13:49:38.141752 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:49:38.141722 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60425df0_fb04_4d4b_92bf_ede71cdf72b9.slice/crio-4dfa577534743c20c34b1c634c71c060ad5514877b7809b36b835f0136916c42 WatchSource:0}: Error finding container 4dfa577534743c20c34b1c634c71c060ad5514877b7809b36b835f0136916c42: Status 404 returned error can't find the container with id 4dfa577534743c20c34b1c634c71c060ad5514877b7809b36b835f0136916c42 Jul 06 13:49:38.791402 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:38.791361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" event={"ID":"60425df0-fb04-4d4b-92bf-ede71cdf72b9","Type":"ContainerStarted","Data":"4dfa577534743c20c34b1c634c71c060ad5514877b7809b36b835f0136916c42"} Jul 06 13:49:39.007515 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:39.007479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:49:39.798620 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:39.798584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" event={"ID":"60425df0-fb04-4d4b-92bf-ede71cdf72b9","Type":"ContainerStarted","Data":"2e12ea2ece99318b7f61544a30cb76b1c8d63dce0d0aa18367a16f52f6cea4a2"} Jul 06 13:49:39.822131 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:39.822069 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" podStartSLOduration=1.745946804 podStartE2EDuration="2.82204752s" totalImagesPullingTime="1.076100716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:49:39.820224779 +0000 UTC m=+595.570038344" watchObservedRunningTime="2026-07-06 13:49:39.82204752 +0000 UTC m=+595.571861088" Jul 06 13:49:41.258842 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:41.258781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:44.642190 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:44.642150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:44.743019 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:44.742983 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:47.603140 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:47.603103 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:49:48.000107 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:48.000006 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:48.000107 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:48.000070 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:48.012846 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:48.012820 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:48.856457 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:48.856430 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh" Jul 06 13:49:56.699664 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.699624 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz"] Jul 06 13:49:56.706008 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.705992 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.708850 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.708829 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 13:49:56.712708 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.712685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz"] Jul 06 13:49:56.720099 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.720079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:49:56.787528 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.787488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.787528 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.787527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.787737 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.787554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.787737 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.787577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.787737 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.787608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmp62\" (UniqueName: \"kubernetes.io/projected/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-kube-api-access-hmp62\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.787737 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.787657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888065 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888065 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888320 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888320 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888320 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmp62\" (UniqueName: \"kubernetes.io/projected/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-kube-api-access-hmp62\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888320 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888547 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888457 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888547 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888505 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.888673 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.888653 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.890519 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.890492 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.891135 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.891113 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:56.896717 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:56.896690 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmp62\" (UniqueName: \"kubernetes.io/projected/7bdcfd45-1368-4bb4-8853-7e9f34e7d694-kube-api-access-hmp62\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz\" (UID: \"7bdcfd45-1368-4bb4-8853-7e9f34e7d694\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:57.016472 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:57.016381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:49:57.152161 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:57.152135 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz"] Jul 06 13:49:57.153727 ip-10-0-134-202 kubenswrapper[2562]: W0706 13:49:57.153700 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bdcfd45_1368_4bb4_8853_7e9f34e7d694.slice/crio-9fd24458992b117e57d23a39bb431c6ba11cfb48d2407b47b93e6d445b13371b WatchSource:0}: Error finding container 9fd24458992b117e57d23a39bb431c6ba11cfb48d2407b47b93e6d445b13371b: Status 404 returned error can't find the container with id 9fd24458992b117e57d23a39bb431c6ba11cfb48d2407b47b93e6d445b13371b Jul 06 13:49:57.155528 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:57.155512 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 13:49:57.901347 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:57.901302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" event={"ID":"7bdcfd45-1368-4bb4-8853-7e9f34e7d694","Type":"ContainerStarted","Data":"f0f078fd1170d5da88c8b9ed0322dd2cc816ccfc54f5c67ac9dfd60376a59703"} Jul 06 13:49:57.901704 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:57.901355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" event={"ID":"7bdcfd45-1368-4bb4-8853-7e9f34e7d694","Type":"ContainerStarted","Data":"9fd24458992b117e57d23a39bb431c6ba11cfb48d2407b47b93e6d445b13371b"} Jul 06 13:49:57.930508 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:57.930448 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" podStartSLOduration=1.704191282 podStartE2EDuration="1.930430738s" totalImagesPullingTime="226.239456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 13:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 13:49:57.927820098 +0000 UTC m=+613.677633658" watchObservedRunningTime="2026-07-06 13:49:57.930430738 +0000 UTC m=+613.680244302" Jul 06 13:49:58.910731 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:49:58.910695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:50:07.017076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:07.016985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:50:07.017076 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:07.017044 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:50:07.029861 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:07.029831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:50:07.967116 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:07.967086 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz" Jul 06 13:50:13.371701 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:13.371670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:13.428625 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:13.428593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:29.770529 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:29.770491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:29.868931 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:29.868894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:31.036670 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:31.036637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:31.125129 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:31.125098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:46.685986 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:46.685935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:46.783083 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:46.783050 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:50:59.208158 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:50:59.208122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:51:04.647251 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:04.647215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:51:10.206204 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:10.206162 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:51:21.001205 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:21.001168 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:51:30.244603 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:30.244568 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:51:40.415205 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:40.415126 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:51:43.514802 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:43.514766 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:51:43.709940 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:43.709900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:51:44.776434 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:44.776397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:51:52.561074 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:52.561039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:51:55.514017 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:51:55.513974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:08.935968 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:08.935931 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:52:21.205523 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:21.205481 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:52:26.264498 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:26.264464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:26.355018 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:26.354984 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:26.511772 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:26.511728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:33.333152 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:33.333104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:33.441343 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:33.441313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:33.533963 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:33.533929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:33.758557 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:33.758464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:40.398473 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:40.398439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:40.459881 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:40.459852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:52:56.498767 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:52:56.498723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:53:04.001696 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:53:04.001661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:53:15.131917 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:53:15.131831 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:53:22.903042 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:53:22.902991 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:53:32.602216 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:53:32.602178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:53:43.308870 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:53:43.308830 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:53:53.596105 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:53:53.596067 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:54:02.903673 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:54:02.903628 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:54:10.108645 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:54:10.108606 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:54:17.794218 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:54:17.794176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:54:28.508935 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:54:28.508899 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:54:37.312861 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:54:37.312752 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:55:02.805557 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:02.805518 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:55:14.320518 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:14.320486 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:55:28.709240 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:28.709203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:55:38.743268 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:38.743236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:55:39.215424 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:39.215391 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:55:40.319443 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:40.319400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:55:50.553968 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:50.553931 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:55:59.306153 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:55:59.306117 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:56:07.900398 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:56:07.900309 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:56:45.165864 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:56:45.165824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:57:57.512567 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:57:57.512481 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:58:06.300037 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:58:06.299999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:58:21.096683 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:58:21.096646 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:58:31.104398 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:58:31.104358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:58:42.441772 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:58:42.441724 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:58:52.503449 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:58:52.503410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:59:02.104405 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:59:02.104366 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:59:07.805134 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:59:07.805034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 13:59:13.004142 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:59:13.004098 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:59:23.803971 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:59:23.803933 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:59:33.308480 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:59:33.308446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 13:59:43.414634 ip-10-0-134-202 kubenswrapper[2562]: I0706 13:59:43.414596 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:00:08.503326 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:00:08.503284 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:00:15.810543 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:00:15.810509 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:00:24.631066 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:00:24.631024 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:00:36.212373 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:00:36.212277 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:00:50.916273 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:00:50.916229 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:00.000556 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:00.000521 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:04.694433 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:04.694394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:01:05.895059 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:05.895016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:01:09.104364 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:09.104327 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:19.116852 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:19.116802 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:27.411641 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:27.411601 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:36.201552 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:36.201517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:49.700815 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:49.700769 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:01:58.408284 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:01:58.408245 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:02:09.304193 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:02:09.304097 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:02:20.801880 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:02:20.801848 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:02:32.506362 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:02:32.506324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:02:41.016573 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:02:41.016536 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:02:52.232070 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:02:52.232030 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:03:04.503350 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:04.503313 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:03:15.505359 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:15.505316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:03:20.988192 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:20.988153 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:21.064113 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:21.064082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:21.432426 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:21.432394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:27.653110 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:27.653070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:27.753072 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:27.753040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:27.891072 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:27.891038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:27.982841 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:27.982741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:29.132227 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:29.132192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:29.207327 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:29.207294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:29.327397 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:29.327368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:29.629937 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:29.629891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:29.814485 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:29.814449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:29.916037 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:29.915945 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:31.808093 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:31.808060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:34.347471 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:34.347427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:34.575942 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:34.575899 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:34.675986 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:34.675902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:34.756370 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:34.756338 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:36.445210 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:36.445177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:40.513484 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:40.513451 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:40.614754 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:40.614720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:40.749486 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:40.749451 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:40.846438 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:40.846406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:41.976843 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:41.976780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:42.080603 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:42.080570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:42.165391 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:42.165357 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:42.415053 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:42.415020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:42.652488 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:42.652454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:42.752263 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:42.752186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:44.638673 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:44.638640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:47.296062 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:47.296023 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:47.609736 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:47.609651 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:53.711297 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:53.711263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:53.821888 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:53.821853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:03:55.853970 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:03:55.853933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:00.359503 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:00.359464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:00.435656 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:00.435623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:00.631171 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:00.631074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:00.718107 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:00.718074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:01.805346 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:01.805313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:01.876561 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:01.876524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:01.935330 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:01.935302 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:02.341050 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:02.341016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:02.444852 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:02.444821 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:02.539560 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:02.539524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:04.411042 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:04.411007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:06.404275 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:06.404247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:06.548699 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:06.548659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:06.669530 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:06.669443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:06.843824 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:06.843772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:08.655698 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:08.655667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:13.012770 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:13.012737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:13.120516 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:13.120479 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:13.503090 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:13.503052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:13.596671 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:13.596636 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:13.695445 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:13.695411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:15.120761 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:15.120721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:21.417415 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:21.417385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:21.526759 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:21.526725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:21.608866 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:21.608838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:21.707540 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:21.707467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:22.128945 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:22.128914 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:38.849802 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:38.849751 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal" podUID="4b12aea9bf8b99530f8679e8c6d1d16c" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Jul 06 14:04:49.636967 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:49.636935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:54.523951 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:54.523917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:54.624027 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:54.623991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:54.724950 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:54.724915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:04:56.592211 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:04:56.592177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:10.621811 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:10.621757 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:16.770856 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:16.770819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:16.898229 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:16.898188 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:16.994271 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:16.994232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:25.350496 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:25.350460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:27.556840 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:27.556779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:28.366415 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:28.366378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:34.814616 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:34.814578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:34.916087 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:34.916055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:35.014552 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:35.014511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:35.115069 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:35.114987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:35.899407 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:35.899372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:36.003019 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:36.002985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:36.380999 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:36.380960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:37.208502 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:37.208461 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:05:37.752481 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:37.752446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:44.346437 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:44.346398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:44.415893 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:44.415863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:44.826529 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:44.826494 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:05:45.053162 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:45.053128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:51.448174 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:51.448129 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:51.543297 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:51.543258 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:51.716319 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:51.716230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:52.061176 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:52.061144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:52.195728 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:52.195695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:05:52.597225 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:52.597189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:55.177167 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:55.177134 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:55.283060 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:55.283030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:55.376600 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:55.376565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:55.675297 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:55.675264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:56.007210 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:56.007117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:56.107666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:56.107637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:05:59.610856 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:59.610825 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:05:59.693059 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:05:59.693029 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:00.399527 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:00.399485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:00.498917 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:00.498892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:00.826632 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:00.826602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:06.787228 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:06.787194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:09.907129 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:09.907094 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:10.703452 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:10.703419 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:06:14.158219 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:14.158188 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:14.216223 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:14.216189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:14.441161 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:14.441078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:14.690346 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:14.690310 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:14.804040 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:14.804006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:14.900978 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:14.900947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:15.024540 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:15.024509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:21.347203 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:21.347165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:21.408276 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:21.408245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:21.560051 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:21.560016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:21.630909 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:21.630828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:21.730474 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:21.730442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:28.201991 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:28.201955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:28.263947 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:28.263914 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:37.956733 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:37.956696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:38.057171 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:38.057138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:40.941810 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:40.941753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:41.009609 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:41.009568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:41.083309 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:41.083267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:41.418128 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:41.418095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:41.629244 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:41.629214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:41.730048 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:41.729961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:44.808809 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:44.808759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:06:45.424947 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:45.424918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:50.736629 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:50.736585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:06:54.806728 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:06:54.806688 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:07:06.602559 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:06.602522 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:07:07.586436 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:07.586400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:07.687399 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:07.687368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:13.306765 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:13.306733 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:13.840438 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:13.840408 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:13.942003 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:13.941970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:14.088999 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:14.088969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:17.003570 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:17.003532 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:07:19.345972 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:19.345938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:23.110505 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:23.110466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:23.187928 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:23.187892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:24.430474 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:24.430435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:07:29.006060 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:07:29.006022 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:08:00.664515 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:00.664480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:02.145620 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:02.145586 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:03.603564 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:03.603531 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:08:08.396123 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:08.396027 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:08.457435 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:08.457402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:08.675904 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:08.675828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:08.919438 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:08.919407 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:09.971051 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:09.970995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:10.069689 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:10.069659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:10.169539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:10.169505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:10.917988 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:10.917951 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:11.015891 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:11.015856 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:11.172680 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:11.172600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:12.513992 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:12.513960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:12.906697 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:12.906666 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:08:15.019036 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:15.019001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:19.208239 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:19.208208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:19.309488 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:19.309453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:19.921638 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:19.921604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:24.202928 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:24.202896 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:08:26.230623 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:26.230571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:26.334430 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:26.334389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:26.433343 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:26.433310 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:36.954706 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:36.954671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:37.313570 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:37.313541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:08:58.289457 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:08:58.289420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:04.454270 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:04.454233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:04.554106 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:04.554068 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:04.831769 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:04.831736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:05.181568 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:05.181493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:05.807690 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:05.807656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:07.997937 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:07.997902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:08.079125 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:08.079092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:08.159884 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:08.159854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:08.535902 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:08.535872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:08.637043 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:08.637003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:08.745979 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:08.745943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:12.391059 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:12.391026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:12.746474 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:12.746383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:13.202033 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:13.202005 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:13.327756 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:13.327718 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:09:13.657485 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:13.657455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:16.002836 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:16.002773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:16.098492 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:16.098456 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:16.423745 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:16.423705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:16.971534 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:16.971502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:17.221036 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:17.221006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:17.323665 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:17.323631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:20.428194 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:20.428152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:21.927149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:21.927115 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:09:29.341912 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:29.341879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:29.442569 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:29.442532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:31.422725 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.422689 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv"] Jul 06 14:09:31.426127 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.426111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.430344 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.430317 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23d6fa\"/\"default-dockercfg-54dsq\"" Jul 06 14:09:31.430477 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.430321 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23d6fa\"/\"test-model-23d6fa-kserve-self-signed-certs\"" Jul 06 14:09:31.430477 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.430328 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23d6fa\"/\"kube-root-ca.crt\"" Jul 06 14:09:31.430477 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.430349 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23d6fa\"/\"openshift-service-ca.crt\"" Jul 06 14:09:31.436170 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.436148 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv"] Jul 06 14:09:31.541964 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.541928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-home\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.542144 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.541968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdb9\" (UniqueName: \"kubernetes.io/projected/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-kube-api-access-9zdb9\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.542144 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.542014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-dshm\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.542144 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.542045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tmp-dir\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.542144 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.542092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tls-certs\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.542309 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.542146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-model-cache\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.596231 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.596199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:31.643273 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-home\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643273 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdb9\" (UniqueName: \"kubernetes.io/projected/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-kube-api-access-9zdb9\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643491 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643295 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-dshm\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643491 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643313 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tmp-dir\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643491 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tls-certs\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643491 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-model-cache\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643677 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643656 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-home\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643838 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643816 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tmp-dir\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.643909 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.643827 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-model-cache\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.645617 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.645596 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-dshm\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.645890 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.645872 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tls-certs\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.653378 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.653347 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdb9\" (UniqueName: \"kubernetes.io/projected/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-kube-api-access-9zdb9\") pod \"test-model-23d6fa-kserve-8597f66549-g4msv\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.737212 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.737116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:31.870956 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:31.870930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv"] Jul 06 14:09:31.872628 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:09:31.872604 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d475f6_0f26_4edc_ba14_8c1bffb2047a.slice/crio-ac2e086c109c9120b9087952037569dbaae65bd3d290f650154bef4082ed0595 WatchSource:0}: Error finding container ac2e086c109c9120b9087952037569dbaae65bd3d290f650154bef4082ed0595: Status 404 returned error can't find the container with id ac2e086c109c9120b9087952037569dbaae65bd3d290f650154bef4082ed0595 Jul 06 14:09:32.097516 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:32.097478 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" event={"ID":"a0d475f6-0f26-4edc-ba14-8c1bffb2047a","Type":"ContainerStarted","Data":"74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b"} Jul 06 14:09:32.097516 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:32.097517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" event={"ID":"a0d475f6-0f26-4edc-ba14-8c1bffb2047a","Type":"ContainerStarted","Data":"ac2e086c109c9120b9087952037569dbaae65bd3d290f650154bef4082ed0595"} Jul 06 14:09:32.115977 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:32.115930 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" podStartSLOduration=1.115914977 podStartE2EDuration="1.115914977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:09:32.115446242 +0000 UTC m=+1787.865259836" watchObservedRunningTime="2026-07-06 14:09:32.115914977 +0000 UTC m=+1787.865728544" Jul 06 14:09:34.798412 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:34.798374 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:09:38.117322 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:38.117233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:38.218392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:38.218364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:09:41.738057 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:41.738013 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:41.738616 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:41.738074 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:41.750725 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:41.750700 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:42.167171 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:42.167138 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:09:47.503120 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:09:47.503083 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:10:07.029799 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:07.029755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:10.407760 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:10.407726 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:10:24.086080 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:24.086044 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv"] Jul 06 14:10:24.104041 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:24.104008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:25.023282 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:25.023252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:25.221407 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:25.221373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:25.633130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:25.633097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:25.730757 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:25.730728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:25.849169 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:25.849138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:25.922697 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:25.922613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:28.200289 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:28.200256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:10:30.685842 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:30.685808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:10:39.106478 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.106426 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" podUID="a0d475f6-0f26-4edc-ba14-8c1bffb2047a" containerName="main" containerID="cri-o://74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b" gracePeriod=45 Jul 06 14:10:39.364844 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.364747 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:10:39.426982 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.426954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tmp-dir\") pod \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " Jul 06 14:10:39.426982 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.426988 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tls-certs\") pod \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " Jul 06 14:10:39.427228 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427034 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-home\") pod \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " Jul 06 14:10:39.427228 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zdb9\" (UniqueName: \"kubernetes.io/projected/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-kube-api-access-9zdb9\") pod \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " Jul 06 14:10:39.427228 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427104 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-model-cache\") pod \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " Jul 06 14:10:39.427228 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427128 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-dshm\") pod \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\" (UID: \"a0d475f6-0f26-4edc-ba14-8c1bffb2047a\") " Jul 06 14:10:39.427422 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427275 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0d475f6-0f26-4edc-ba14-8c1bffb2047a" (UID: "a0d475f6-0f26-4edc-ba14-8c1bffb2047a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:10:39.427422 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427286 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-home" (OuterVolumeSpecName: "home") pod "a0d475f6-0f26-4edc-ba14-8c1bffb2047a" (UID: "a0d475f6-0f26-4edc-ba14-8c1bffb2047a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:10:39.427422 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427330 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0d475f6-0f26-4edc-ba14-8c1bffb2047a" (UID: "a0d475f6-0f26-4edc-ba14-8c1bffb2047a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:10:39.427540 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427425 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:10:39.427540 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.427446 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:10:39.429326 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.429296 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-dshm" (OuterVolumeSpecName: "dshm") pod "a0d475f6-0f26-4edc-ba14-8c1bffb2047a" (UID: "a0d475f6-0f26-4edc-ba14-8c1bffb2047a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:10:39.429326 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.429307 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-kube-api-access-9zdb9" (OuterVolumeSpecName: "kube-api-access-9zdb9") pod "a0d475f6-0f26-4edc-ba14-8c1bffb2047a" (UID: "a0d475f6-0f26-4edc-ba14-8c1bffb2047a"). InnerVolumeSpecName "kube-api-access-9zdb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:10:39.429326 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.429320 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0d475f6-0f26-4edc-ba14-8c1bffb2047a" (UID: "a0d475f6-0f26-4edc-ba14-8c1bffb2047a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:10:39.458474 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.458443 2562 generic.go:356] "Generic (PLEG): container finished" podID="a0d475f6-0f26-4edc-ba14-8c1bffb2047a" containerID="74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b" exitCode=0 Jul 06 14:10:39.458636 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.458485 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" Jul 06 14:10:39.458636 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.458535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" event={"ID":"a0d475f6-0f26-4edc-ba14-8c1bffb2047a","Type":"ContainerDied","Data":"74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b"} Jul 06 14:10:39.458636 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.458590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv" event={"ID":"a0d475f6-0f26-4edc-ba14-8c1bffb2047a","Type":"ContainerDied","Data":"ac2e086c109c9120b9087952037569dbaae65bd3d290f650154bef4082ed0595"} Jul 06 14:10:39.458636 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.458614 2562 scope.go:117] "RemoveContainer" containerID="74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b" Jul 06 14:10:39.467624 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.467609 2562 scope.go:117] "RemoveContainer" containerID="74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b" Jul 06 14:10:39.467944 ip-10-0-134-202 kubenswrapper[2562]: E0706 14:10:39.467922 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b\": container with ID starting with 74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b not found: ID does not exist" containerID="74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b" Jul 06 14:10:39.468012 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.467954 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b"} err="failed to get container status \"74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b\": rpc error: code = NotFound desc = could not find container \"74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b\": container with ID starting with 74931c7f4dd72ba066d94daa0bcb1b0d86a3ff7ea7b74fdba5862bff9c92f63b not found: ID does not exist" Jul 06 14:10:39.510027 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.509994 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv"] Jul 06 14:10:39.515145 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.515119 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-23d6fa/test-model-23d6fa-kserve-8597f66549-g4msv"] Jul 06 14:10:39.528372 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.528353 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdb9\" (UniqueName: \"kubernetes.io/projected/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-kube-api-access-9zdb9\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:10:39.528442 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.528375 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:10:39.528442 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.528386 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:10:39.528442 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.528393 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d475f6-0f26-4edc-ba14-8c1bffb2047a-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:10:39.543580 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:39.543550 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d475f6-0f26-4edc-ba14-8c1bffb2047a" path="/var/lib/kubelet/pods/a0d475f6-0f26-4edc-ba14-8c1bffb2047a/volumes" Jul 06 14:10:40.998600 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:10:40.998564 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:11:00.875890 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:00.875847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:01.188776 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:01.188699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:01.396750 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:01.396723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:01.586864 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:01.586828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:02.016893 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:02.016800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:02.268008 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:02.267917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:02.371522 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:02.371486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:02.532100 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:02.532006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:05.011530 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:05.011495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:11:08.564351 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:08.564272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:08.626271 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:08.626234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:08.847066 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:08.846973 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:09.058027 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:09.057969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:09.389705 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:09.389663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:09.725007 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:09.724913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:09.923700 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:09.923671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:12.636861 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:12.636753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:12.767580 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:12.767547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:12.933090 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:12.933007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:13.038465 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:13.038434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:14.055336 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:14.055300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:14.254438 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:14.254406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:17.054416 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:17.054370 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:17.512926 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:17.512828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:17.819239 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:17.819207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:18.109171 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:18.109086 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:11:18.192765 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:18.192735 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:20.567248 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:20.567219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:20.660480 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:20.660448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:20.733167 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:20.733135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:21.190729 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:21.190697 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:21.301359 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:21.301323 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:21.400519 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:21.400459 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:25.055001 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:25.054968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:27.805368 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:27.805333 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:11:33.230530 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:33.230487 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:33.619616 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:33.619577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:33.720247 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:33.720204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:33.820465 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:33.820430 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:33.919630 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:33.919519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:39.506310 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:39.506270 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:11:40.104967 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:40.104932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:40.205087 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:40.205055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:40.386738 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:40.386654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:40.769275 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:40.769162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:40.880894 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:40.880857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:41.225346 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:41.225319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:41.330411 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:41.330377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:41.426218 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:41.426154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:41.575647 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:41.575614 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:47.738094 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:47.738056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:47.808847 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:47.808814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:48.010038 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:48.009936 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:48.304007 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:48.303978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:48.649527 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:48.649442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:48.965971 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:48.965890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:49.453051 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:49.452985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:50.998937 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:50.998900 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:11:51.809419 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:51.809387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:51.894660 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:51.894630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:51.977781 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:51.977750 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:52.392034 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:52.391990 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:52.486277 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:52.486247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:56.355612 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:56.355582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:56.763299 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:56.763220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:57.089834 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:57.089798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:11:57.858573 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:11:57.858544 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:00.004207 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:00.004172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:00.091085 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:00.091058 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:00.269534 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:00.269453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:00.584894 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:00.584858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:00.770896 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:00.770863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:00.870669 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:00.870582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:01.404725 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:01.404690 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:12:04.497024 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:04.496980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:08.337348 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.334709 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6"] Jul 06 14:12:08.337348 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.335515 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d475f6-0f26-4edc-ba14-8c1bffb2047a" containerName="main" Jul 06 14:12:08.337348 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.335537 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d475f6-0f26-4edc-ba14-8c1bffb2047a" containerName="main" Jul 06 14:12:08.337348 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.335757 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d475f6-0f26-4edc-ba14-8c1bffb2047a" containerName="main" Jul 06 14:12:08.339844 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.339820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.349779 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.349749 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"kube-root-ca.crt\"" Jul 06 14:12:08.350023 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.349993 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"openshift-service-ca.crt\"" Jul 06 14:12:08.350112 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.349816 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"auth-test-model-973328-kserve-self-signed-certs\"" Jul 06 14:12:08.350231 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.349877 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"default-dockercfg-87hpn\"" Jul 06 14:12:08.354039 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.353734 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6"] Jul 06 14:12:08.455524 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.455487 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:08.503211 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.503175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-home\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.503392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.503231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tls-certs\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.503392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.503290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d828h\" (UniqueName: \"kubernetes.io/projected/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-kube-api-access-d828h\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.503392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.503333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-model-cache\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.503392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.503374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-dshm\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.503554 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.503394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tmp-dir\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.553239 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.553209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:08.604715 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.604624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tls-certs\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.604715 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.604669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d828h\" (UniqueName: \"kubernetes.io/projected/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-kube-api-access-d828h\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.604923 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.604826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-model-cache\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.604967 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.604916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-dshm\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.604967 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.604946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tmp-dir\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.605048 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.604989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-home\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.605225 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.605195 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-model-cache\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.605320 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.605272 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tmp-dir\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.605320 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.605308 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-home\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.607118 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.607094 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-dshm\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.607279 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.607264 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tls-certs\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.612883 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.612862 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d828h\" (UniqueName: \"kubernetes.io/projected/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-kube-api-access-d828h\") pod \"auth-test-model-973328-kserve-65cfcf64f9-p65k6\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.655958 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.655920 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:08.656567 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.656543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:08.793534 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.793498 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6"] Jul 06 14:12:08.795283 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:12:08.795243 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec11f3c1_6d97_470b_b2d1_e29e0205b65e.slice/crio-c337ba11aa9650e7b11f4a76d32944d7fce15b84003c0b357630922910559821 WatchSource:0}: Error finding container c337ba11aa9650e7b11f4a76d32944d7fce15b84003c0b357630922910559821: Status 404 returned error can't find the container with id c337ba11aa9650e7b11f4a76d32944d7fce15b84003c0b357630922910559821 Jul 06 14:12:08.934506 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.934425 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" event={"ID":"ec11f3c1-6d97-470b-b2d1-e29e0205b65e","Type":"ContainerStarted","Data":"9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca"} Jul 06 14:12:08.934506 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.934465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" event={"ID":"ec11f3c1-6d97-470b-b2d1-e29e0205b65e","Type":"ContainerStarted","Data":"c337ba11aa9650e7b11f4a76d32944d7fce15b84003c0b357630922910559821"} Jul 06 14:12:08.955224 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:08.955174 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" podStartSLOduration=0.955158549 podStartE2EDuration="955.158549ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:12:08.953869212 +0000 UTC m=+1944.703682778" watchObservedRunningTime="2026-07-06 14:12:08.955158549 +0000 UTC m=+1944.704972114" Jul 06 14:12:09.625310 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:09.625274 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:12:13.195650 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:13.195619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:13.298329 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:13.298290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:18.657112 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:18.657071 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:18.657112 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:18.657117 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:18.669646 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:18.669619 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:18.717340 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:18.717307 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:12:19.004080 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:19.003999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:12:21.578200 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:21.578170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:21.680131 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:21.680100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:40.188803 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:40.188748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:12:43.606762 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:12:43.606728 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:13:03.015780 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:13:03.015742 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:13:17.901745 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:13:17.901705 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:13:30.896046 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:13:30.896008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:13:35.580899 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:13:35.580864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:13:48.007804 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:13:48.007756 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:14:05.597011 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:05.596980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:14:05.611334 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:05.611304 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:14:21.314956 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:21.314876 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:14:33.485008 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:33.484970 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6"] Jul 06 14:14:33.498591 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:33.498559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:14:34.381738 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:34.381705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:14:36.895373 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:36.895333 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:14:41.873010 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:41.872972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:14:46.613897 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.613861 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b"] Jul 06 14:14:46.617163 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.617145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.620227 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.620203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"sub-test-model-973328-kserve-self-signed-certs\"" Jul 06 14:14:46.627290 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.627268 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b"] Jul 06 14:14:46.648350 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.648320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:14:46.701232 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.701194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-model-cache\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.701427 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.701240 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-home\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.701427 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.701263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tmp-dir\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.701427 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.701314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tls-certs\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.701427 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.701336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n962\" (UniqueName: \"kubernetes.io/projected/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-kube-api-access-4n962\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.701427 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.701414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-dshm\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.801972 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.801930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tls-certs\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.801972 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.801974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n962\" (UniqueName: \"kubernetes.io/projected/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-kube-api-access-4n962\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802257 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-dshm\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802257 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-model-cache\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802257 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-home\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802427 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tmp-dir\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802599 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802575 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-model-cache\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802608 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-home\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.802723 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.802676 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tmp-dir\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.804371 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.804349 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-dshm\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.804540 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.804521 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tls-certs\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.814392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.814370 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n962\" (UniqueName: \"kubernetes.io/projected/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-kube-api-access-4n962\") pod \"sub-test-model-973328-kserve-798bf67fdc-9rd6b\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:46.927111 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:46.927024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:47.078048 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:47.078023 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b"] Jul 06 14:14:47.080086 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:14:47.080062 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ebe96c_5115_46f2_af5b_bb2ef901cdbf.slice/crio-c6b08299d7d20d70f82a30dd8103f7c98b1222f1a6cd3cb1b7c654f4b2b07fdd WatchSource:0}: Error finding container c6b08299d7d20d70f82a30dd8103f7c98b1222f1a6cd3cb1b7c654f4b2b07fdd: Status 404 returned error can't find the container with id c6b08299d7d20d70f82a30dd8103f7c98b1222f1a6cd3cb1b7c654f4b2b07fdd Jul 06 14:14:47.768994 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:47.768961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" event={"ID":"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf","Type":"ContainerStarted","Data":"94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a"} Jul 06 14:14:47.768994 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:47.769005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" event={"ID":"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf","Type":"ContainerStarted","Data":"c6b08299d7d20d70f82a30dd8103f7c98b1222f1a6cd3cb1b7c654f4b2b07fdd"} Jul 06 14:14:47.789521 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:47.789467 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" podStartSLOduration=1.789451632 podStartE2EDuration="1.789451632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:14:47.78790943 +0000 UTC m=+2103.537723022" watchObservedRunningTime="2026-07-06 14:14:47.789451632 +0000 UTC m=+2103.539265197" Jul 06 14:14:48.513254 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.513206 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" podUID="ec11f3c1-6d97-470b-b2d1-e29e0205b65e" containerName="main" containerID="cri-o://9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca" gracePeriod=45 Jul 06 14:14:48.766890 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.766817 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:14:48.775867 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.775834 2562 generic.go:356] "Generic (PLEG): container finished" podID="ec11f3c1-6d97-470b-b2d1-e29e0205b65e" containerID="9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca" exitCode=0 Jul 06 14:14:48.776230 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.775886 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" Jul 06 14:14:48.776230 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.775918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" event={"ID":"ec11f3c1-6d97-470b-b2d1-e29e0205b65e","Type":"ContainerDied","Data":"9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca"} Jul 06 14:14:48.776230 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.775960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6" event={"ID":"ec11f3c1-6d97-470b-b2d1-e29e0205b65e","Type":"ContainerDied","Data":"c337ba11aa9650e7b11f4a76d32944d7fce15b84003c0b357630922910559821"} Jul 06 14:14:48.776230 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.775977 2562 scope.go:117] "RemoveContainer" containerID="9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca" Jul 06 14:14:48.785809 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.785241 2562 scope.go:117] "RemoveContainer" containerID="9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca" Jul 06 14:14:48.785809 ip-10-0-134-202 kubenswrapper[2562]: E0706 14:14:48.785574 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca\": container with ID starting with 9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca not found: ID does not exist" containerID="9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca" Jul 06 14:14:48.785809 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.785602 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca"} err="failed to get container status \"9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca\": rpc error: code = NotFound desc = could not find container \"9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca\": container with ID starting with 9afa240dac8aeba65863f142b0975478461d83dd8769c096b71a373fdfa167ca not found: ID does not exist" Jul 06 14:14:48.923177 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923144 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-home\") pod \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " Jul 06 14:14:48.923387 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923197 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tls-certs\") pod \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " Jul 06 14:14:48.923387 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tmp-dir\") pod \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " Jul 06 14:14:48.923387 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923281 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d828h\" (UniqueName: \"kubernetes.io/projected/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-kube-api-access-d828h\") pod \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " Jul 06 14:14:48.923387 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923305 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-model-cache\") pod \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " Jul 06 14:14:48.923387 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923332 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-dshm\") pod \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\" (UID: \"ec11f3c1-6d97-470b-b2d1-e29e0205b65e\") " Jul 06 14:14:48.923640 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923480 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-home" (OuterVolumeSpecName: "home") pod "ec11f3c1-6d97-470b-b2d1-e29e0205b65e" (UID: "ec11f3c1-6d97-470b-b2d1-e29e0205b65e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:14:48.923640 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923576 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec11f3c1-6d97-470b-b2d1-e29e0205b65e" (UID: "ec11f3c1-6d97-470b-b2d1-e29e0205b65e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:14:48.923640 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923587 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec11f3c1-6d97-470b-b2d1-e29e0205b65e" (UID: "ec11f3c1-6d97-470b-b2d1-e29e0205b65e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:14:48.923880 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923775 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:14:48.923880 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923821 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:14:48.923880 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.923837 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:14:48.925616 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.925588 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-dshm" (OuterVolumeSpecName: "dshm") pod "ec11f3c1-6d97-470b-b2d1-e29e0205b65e" (UID: "ec11f3c1-6d97-470b-b2d1-e29e0205b65e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:14:48.925952 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.925934 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec11f3c1-6d97-470b-b2d1-e29e0205b65e" (UID: "ec11f3c1-6d97-470b-b2d1-e29e0205b65e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:14:48.926029 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:48.926008 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-kube-api-access-d828h" (OuterVolumeSpecName: "kube-api-access-d828h") pod "ec11f3c1-6d97-470b-b2d1-e29e0205b65e" (UID: "ec11f3c1-6d97-470b-b2d1-e29e0205b65e"). InnerVolumeSpecName "kube-api-access-d828h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:14:49.024702 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.024606 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:14:49.024702 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.024641 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d828h\" (UniqueName: \"kubernetes.io/projected/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-kube-api-access-d828h\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:14:49.024702 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.024654 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec11f3c1-6d97-470b-b2d1-e29e0205b65e-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:14:49.098186 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.098151 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6"] Jul 06 14:14:49.101914 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.101892 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-973328/auth-test-model-973328-kserve-65cfcf64f9-p65k6"] Jul 06 14:14:49.496758 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.496727 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:14:49.550406 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:49.550364 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec11f3c1-6d97-470b-b2d1-e29e0205b65e" path="/var/lib/kubelet/pods/ec11f3c1-6d97-470b-b2d1-e29e0205b65e/volumes" Jul 06 14:14:51.299716 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:51.299672 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:14:56.927269 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:56.927234 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:56.927759 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:56.927316 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:56.939774 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:56.939743 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:14:57.838477 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:14:57.838446 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:15:00.133004 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.132967 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722455-46xlw"] Jul 06 14:15:00.133756 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.133393 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec11f3c1-6d97-470b-b2d1-e29e0205b65e" containerName="main" Jul 06 14:15:00.133756 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.133412 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec11f3c1-6d97-470b-b2d1-e29e0205b65e" containerName="main" Jul 06 14:15:00.133756 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.133529 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec11f3c1-6d97-470b-b2d1-e29e0205b65e" containerName="main" Jul 06 14:15:00.138250 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.138232 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:00.140794 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.140766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hq6cd\"" Jul 06 14:15:00.148352 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.148324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722455-46xlw"] Jul 06 14:15:00.226580 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.226530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz64\" (UniqueName: \"kubernetes.io/projected/e8ef975d-07c9-48f6-a9da-9a5b1d31858d-kube-api-access-7gz64\") pod \"maas-api-key-cleanup-29722455-46xlw\" (UID: \"e8ef975d-07c9-48f6-a9da-9a5b1d31858d\") " pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:00.328068 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.328028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz64\" (UniqueName: \"kubernetes.io/projected/e8ef975d-07c9-48f6-a9da-9a5b1d31858d-kube-api-access-7gz64\") pod \"maas-api-key-cleanup-29722455-46xlw\" (UID: \"e8ef975d-07c9-48f6-a9da-9a5b1d31858d\") " pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:00.337694 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.337665 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz64\" (UniqueName: \"kubernetes.io/projected/e8ef975d-07c9-48f6-a9da-9a5b1d31858d-kube-api-access-7gz64\") pod \"maas-api-key-cleanup-29722455-46xlw\" (UID: \"e8ef975d-07c9-48f6-a9da-9a5b1d31858d\") " pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:00.449627 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.449542 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:00.585976 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.585947 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722455-46xlw"] Jul 06 14:15:00.587565 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:15:00.587539 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ef975d_07c9_48f6_a9da_9a5b1d31858d.slice/crio-3be6e8aa92354347e92532ba14d019bf032f4de6ada92a4b585a120aad1ad12b WatchSource:0}: Error finding container 3be6e8aa92354347e92532ba14d019bf032f4de6ada92a4b585a120aad1ad12b: Status 404 returned error can't find the container with id 3be6e8aa92354347e92532ba14d019bf032f4de6ada92a4b585a120aad1ad12b Jul 06 14:15:00.589615 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.589596 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 14:15:00.843471 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:00.843430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" event={"ID":"e8ef975d-07c9-48f6-a9da-9a5b1d31858d","Type":"ContainerStarted","Data":"3be6e8aa92354347e92532ba14d019bf032f4de6ada92a4b585a120aad1ad12b"} Jul 06 14:15:02.855782 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:02.855687 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" event={"ID":"e8ef975d-07c9-48f6-a9da-9a5b1d31858d","Type":"ContainerStarted","Data":"7685eb4f05f2338f8a4dc2dd1379fc2d89a54054fb2244b47ab85c431a910443"} Jul 06 14:15:02.874394 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:02.874340 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" podStartSLOduration=0.870186091 podStartE2EDuration="2.874322239s" totalImagesPullingTime="2.004136148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:15:02.872341811 +0000 UTC m=+2118.622155397" watchObservedRunningTime="2026-07-06 14:15:02.874322239 +0000 UTC m=+2118.624135808" Jul 06 14:15:03.862177 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:03.862143 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8ef975d-07c9-48f6-a9da-9a5b1d31858d" containerID="7685eb4f05f2338f8a4dc2dd1379fc2d89a54054fb2244b47ab85c431a910443" exitCode=0 Jul 06 14:15:03.862568 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:03.862236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" event={"ID":"e8ef975d-07c9-48f6-a9da-9a5b1d31858d","Type":"ContainerDied","Data":"7685eb4f05f2338f8a4dc2dd1379fc2d89a54054fb2244b47ab85c431a910443"} Jul 06 14:15:05.007103 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.007077 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:05.068181 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.068149 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gz64\" (UniqueName: \"kubernetes.io/projected/e8ef975d-07c9-48f6-a9da-9a5b1d31858d-kube-api-access-7gz64\") pod \"e8ef975d-07c9-48f6-a9da-9a5b1d31858d\" (UID: \"e8ef975d-07c9-48f6-a9da-9a5b1d31858d\") " Jul 06 14:15:05.070288 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.070262 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ef975d-07c9-48f6-a9da-9a5b1d31858d-kube-api-access-7gz64" (OuterVolumeSpecName: "kube-api-access-7gz64") pod "e8ef975d-07c9-48f6-a9da-9a5b1d31858d" (UID: "e8ef975d-07c9-48f6-a9da-9a5b1d31858d"). InnerVolumeSpecName "kube-api-access-7gz64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:15:05.169264 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.169172 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz64\" (UniqueName: \"kubernetes.io/projected/e8ef975d-07c9-48f6-a9da-9a5b1d31858d-kube-api-access-7gz64\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:15:05.662972 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.662933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:15:05.873693 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.873611 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" event={"ID":"e8ef975d-07c9-48f6-a9da-9a5b1d31858d","Type":"ContainerDied","Data":"3be6e8aa92354347e92532ba14d019bf032f4de6ada92a4b585a120aad1ad12b"} Jul 06 14:15:05.873693 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.873639 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722455-46xlw" Jul 06 14:15:05.873693 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:05.873657 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be6e8aa92354347e92532ba14d019bf032f4de6ada92a4b585a120aad1ad12b" Jul 06 14:15:07.308642 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:07.308610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:15:10.904989 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:10.904953 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:15:29.317117 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:29.317068 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:15:46.929296 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:46.929261 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:15:51.620819 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:51.620771 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b"] Jul 06 14:15:51.635648 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:51.635618 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:15:52.522654 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:15:52.522618 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:16:03.505629 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:03.505586 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:16:06.640683 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.640638 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" podUID="e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" containerName="main" containerID="cri-o://94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a" gracePeriod=45 Jul 06 14:16:06.891638 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.891581 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:16:06.897745 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.897725 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n962\" (UniqueName: \"kubernetes.io/projected/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-kube-api-access-4n962\") pod \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " Jul 06 14:16:06.897832 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.897765 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tmp-dir\") pod \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " Jul 06 14:16:06.897832 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.897795 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tls-certs\") pod \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " Jul 06 14:16:06.897832 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.897816 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-home\") pod \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " Jul 06 14:16:06.897996 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.897851 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-dshm\") pod \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " Jul 06 14:16:06.897996 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.897869 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-model-cache\") pod \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\" (UID: \"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf\") " Jul 06 14:16:06.898119 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.898025 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" (UID: "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:16:06.898176 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.898150 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:16:06.898176 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.898157 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-home" (OuterVolumeSpecName: "home") pod "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" (UID: "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:16:06.898293 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.898188 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" (UID: "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:16:06.899914 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.899883 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-kube-api-access-4n962" (OuterVolumeSpecName: "kube-api-access-4n962") pod "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" (UID: "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf"). InnerVolumeSpecName "kube-api-access-4n962". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:16:06.900014 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.899947 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" (UID: "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:16:06.900014 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.899984 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-dshm" (OuterVolumeSpecName: "dshm") pod "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" (UID: "e0ebe96c-5115-46f2-af5b-bb2ef901cdbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:16:06.998838 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.998803 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n962\" (UniqueName: \"kubernetes.io/projected/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-kube-api-access-4n962\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:16:06.998838 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.998830 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:16:06.998838 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.998840 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:16:06.998838 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.998847 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:16:06.999109 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:06.998855 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:16:07.193509 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.193422 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" containerID="94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a" exitCode=0 Jul 06 14:16:07.193509 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.193464 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" Jul 06 14:16:07.193710 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.193504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" event={"ID":"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf","Type":"ContainerDied","Data":"94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a"} Jul 06 14:16:07.193710 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.193545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b" event={"ID":"e0ebe96c-5115-46f2-af5b-bb2ef901cdbf","Type":"ContainerDied","Data":"c6b08299d7d20d70f82a30dd8103f7c98b1222f1a6cd3cb1b7c654f4b2b07fdd"} Jul 06 14:16:07.193710 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.193563 2562 scope.go:117] "RemoveContainer" containerID="94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a" Jul 06 14:16:07.203147 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.203131 2562 scope.go:117] "RemoveContainer" containerID="94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a" Jul 06 14:16:07.203389 ip-10-0-134-202 kubenswrapper[2562]: E0706 14:16:07.203371 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a\": container with ID starting with 94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a not found: ID does not exist" containerID="94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a" Jul 06 14:16:07.203439 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.203398 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a"} err="failed to get container status \"94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a\": rpc error: code = NotFound desc = could not find container \"94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a\": container with ID starting with 94bd8e212afa1bc91da7607e3eeeb03b320d8f09f9783e8e281873ed30c9283a not found: ID does not exist" Jul 06 14:16:07.223495 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.223469 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b"] Jul 06 14:16:07.229420 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.229402 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-973328/sub-test-model-973328-kserve-798bf67fdc-9rd6b"] Jul 06 14:16:07.545323 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:07.545293 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" path="/var/lib/kubelet/pods/e0ebe96c-5115-46f2-af5b-bb2ef901cdbf/volumes" Jul 06 14:16:16.206491 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.206444 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n"] Jul 06 14:16:16.207130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.207035 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ef975d-07c9-48f6-a9da-9a5b1d31858d" containerName="cleanup" Jul 06 14:16:16.207130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.207058 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ef975d-07c9-48f6-a9da-9a5b1d31858d" containerName="cleanup" Jul 06 14:16:16.207130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.207075 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" containerName="main" Jul 06 14:16:16.207130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.207083 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" containerName="main" Jul 06 14:16:16.207354 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.207186 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ebe96c-5115-46f2-af5b-bb2ef901cdbf" containerName="main" Jul 06 14:16:16.207354 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.207197 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ef975d-07c9-48f6-a9da-9a5b1d31858d" containerName="cleanup" Jul 06 14:16:16.211448 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.211421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.215537 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.215483 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"rate-test-model-973328-kserve-self-signed-certs\"" Jul 06 14:16:16.215537 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.215488 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"default-dockercfg-87hpn\"" Jul 06 14:16:16.215887 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.215865 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"openshift-service-ca.crt\"" Jul 06 14:16:16.216091 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.216015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-973328\"/\"kube-root-ca.crt\"" Jul 06 14:16:16.218983 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.218930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n"] Jul 06 14:16:16.227990 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.227954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:16:16.277967 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.277937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-tmp-dir\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.278149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.277973 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-home\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.278149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.277992 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-model-cache\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.278149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.278007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-dshm\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.278149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.278099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11444a1c-ba90-445f-a682-bc556aa4e14b-tls-certs\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.278149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.278133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28kt\" (UniqueName: \"kubernetes.io/projected/11444a1c-ba90-445f-a682-bc556aa4e14b-kube-api-access-n28kt\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.378937 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.378893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-tmp-dir\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.378937 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.378942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-home\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379183 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.378971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-model-cache\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379183 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.379000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-dshm\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379183 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.379039 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11444a1c-ba90-445f-a682-bc556aa4e14b-tls-certs\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379183 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.379096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28kt\" (UniqueName: \"kubernetes.io/projected/11444a1c-ba90-445f-a682-bc556aa4e14b-kube-api-access-n28kt\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379404 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.379376 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-tmp-dir\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379472 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.379405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-model-cache\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.379472 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.379439 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-home\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.381397 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.381359 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-dshm\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.381512 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.381491 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11444a1c-ba90-445f-a682-bc556aa4e14b-tls-certs\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.388199 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.388172 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28kt\" (UniqueName: \"kubernetes.io/projected/11444a1c-ba90-445f-a682-bc556aa4e14b-kube-api-access-n28kt\") pod \"rate-test-model-973328-kserve-84794d466c-r6d6n\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.523045 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.522948 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:16.666967 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:16.666939 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n"] Jul 06 14:16:16.668549 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:16:16.668520 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11444a1c_ba90_445f_a682_bc556aa4e14b.slice/crio-c40085222773f82bccfb7e7e591ffe2b12f8f2719941ec2c46e9d65b7894472c WatchSource:0}: Error finding container c40085222773f82bccfb7e7e591ffe2b12f8f2719941ec2c46e9d65b7894472c: Status 404 returned error can't find the container with id c40085222773f82bccfb7e7e591ffe2b12f8f2719941ec2c46e9d65b7894472c Jul 06 14:16:17.247196 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:17.247162 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" event={"ID":"11444a1c-ba90-445f-a682-bc556aa4e14b","Type":"ContainerStarted","Data":"91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f"} Jul 06 14:16:17.247196 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:17.247200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" event={"ID":"11444a1c-ba90-445f-a682-bc556aa4e14b","Type":"ContainerStarted","Data":"c40085222773f82bccfb7e7e591ffe2b12f8f2719941ec2c46e9d65b7894472c"} Jul 06 14:16:17.270503 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:17.270443 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" podStartSLOduration=1.270425865 podStartE2EDuration="1.270425865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:16:17.268219912 +0000 UTC m=+2193.018033477" watchObservedRunningTime="2026-07-06 14:16:17.270425865 +0000 UTC m=+2193.020239459" Jul 06 14:16:19.704487 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:19.704452 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:16:26.523317 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:26.523280 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:26.523915 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:26.523437 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:26.536369 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:26.536340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:27.311383 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:27.311349 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:16:58.709566 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.709527 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2"] Jul 06 14:16:58.713745 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.713721 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.717980 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.717953 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c041\"/\"openshift-service-ca.crt\"" Jul 06 14:16:58.718130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.717986 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c041\"/\"rate-test-model-35c041-kserve-self-signed-certs\"" Jul 06 14:16:58.718130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.718005 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-35c041\"/\"kube-root-ca.crt\"" Jul 06 14:16:58.718130 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.717950 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-35c041\"/\"default-dockercfg-cx4hb\"" Jul 06 14:16:58.721632 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.721609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2"] Jul 06 14:16:58.739715 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.739687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:16:58.750047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.750027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tmp-dir\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.750129 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.750082 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-model-cache\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.750129 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.750124 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-home\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.750199 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.750146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-dshm\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.750199 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.750163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tls-certs\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.750271 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.750202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wmm\" (UniqueName: \"kubernetes.io/projected/f246e29a-393c-44ee-b7f0-f55a183fdf8e-kube-api-access-r9wmm\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851458 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-model-cache\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-home\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-dshm\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tls-certs\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wmm\" (UniqueName: \"kubernetes.io/projected/f246e29a-393c-44ee-b7f0-f55a183fdf8e-kube-api-access-r9wmm\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851666 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tmp-dir\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.851952 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851901 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-model-cache\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.852018 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.851959 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-home\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.852117 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.852100 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tmp-dir\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.853725 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.853706 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-dshm\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.854229 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.854207 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tls-certs\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:58.861308 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:58.861288 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wmm\" (UniqueName: \"kubernetes.io/projected/f246e29a-393c-44ee-b7f0-f55a183fdf8e-kube-api-access-r9wmm\") pod \"rate-test-model-35c041-kserve-667995d5cf-bdpb2\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:59.025185 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:59.025095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:16:59.167286 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:59.167261 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2"] Jul 06 14:16:59.169068 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:16:59.169045 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf246e29a_393c_44ee_b7f0_f55a183fdf8e.slice/crio-cacfe571e32859473829872854c8777616edcefdafb365af757debc1914789bb WatchSource:0}: Error finding container cacfe571e32859473829872854c8777616edcefdafb365af757debc1914789bb: Status 404 returned error can't find the container with id cacfe571e32859473829872854c8777616edcefdafb365af757debc1914789bb Jul 06 14:16:59.471977 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:59.471941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" event={"ID":"f246e29a-393c-44ee-b7f0-f55a183fdf8e","Type":"ContainerStarted","Data":"003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461"} Jul 06 14:16:59.472149 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:59.471986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" event={"ID":"f246e29a-393c-44ee-b7f0-f55a183fdf8e","Type":"ContainerStarted","Data":"cacfe571e32859473829872854c8777616edcefdafb365af757debc1914789bb"} Jul 06 14:16:59.492172 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:16:59.492122 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" podStartSLOduration=1.492105748 podStartE2EDuration="1.492105748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 14:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:16:59.491072122 +0000 UTC m=+2235.240885687" watchObservedRunningTime="2026-07-06 14:16:59.492105748 +0000 UTC m=+2235.241919313" Jul 06 14:17:03.212627 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:03.212589 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:17:09.025997 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:09.025915 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:17:09.025997 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:09.025966 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:17:09.039690 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:09.039661 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:17:09.490646 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:09.490609 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:09.535304 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:09.535277 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:17:09.991812 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:09.991768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:13.942696 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:13.942658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:14.043630 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:14.043597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:22.315473 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:22.315435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:22.418263 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:22.418226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:38.109969 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:38.109937 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:38.175947 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:38.175911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:17:44.098200 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:44.098166 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:17:57.198315 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:17:57.198282 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:18:03.124543 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:03.124508 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n"] Jul 06 14:18:03.143912 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:03.143876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:04.058540 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:04.058503 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2"] Jul 06 14:18:04.123827 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:04.123771 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:05.136543 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:05.136509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:05.520968 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:05.520885 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:05.621734 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:05.621691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:05.721870 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:05.721834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:05.816782 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:05.816750 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:10.204948 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:10.204901 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:18:10.617189 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:10.617135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:18.145015 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.144970 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" podUID="11444a1c-ba90-445f-a682-bc556aa4e14b" containerName="main" containerID="cri-o://91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f" gracePeriod=45 Jul 06 14:18:18.399621 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.399560 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:18:18.573014 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.572978 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-tmp-dir\") pod \"11444a1c-ba90-445f-a682-bc556aa4e14b\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " Jul 06 14:18:18.573214 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n28kt\" (UniqueName: \"kubernetes.io/projected/11444a1c-ba90-445f-a682-bc556aa4e14b-kube-api-access-n28kt\") pod \"11444a1c-ba90-445f-a682-bc556aa4e14b\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " Jul 06 14:18:18.573214 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573064 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-dshm\") pod \"11444a1c-ba90-445f-a682-bc556aa4e14b\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " Jul 06 14:18:18.573214 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573105 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-model-cache\") pod \"11444a1c-ba90-445f-a682-bc556aa4e14b\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " Jul 06 14:18:18.573214 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573125 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-home\") pod \"11444a1c-ba90-445f-a682-bc556aa4e14b\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " Jul 06 14:18:18.573214 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11444a1c-ba90-445f-a682-bc556aa4e14b-tls-certs\") pod \"11444a1c-ba90-445f-a682-bc556aa4e14b\" (UID: \"11444a1c-ba90-445f-a682-bc556aa4e14b\") " Jul 06 14:18:18.573518 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573322 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11444a1c-ba90-445f-a682-bc556aa4e14b" (UID: "11444a1c-ba90-445f-a682-bc556aa4e14b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:18.573518 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573429 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-model-cache" (OuterVolumeSpecName: "model-cache") pod "11444a1c-ba90-445f-a682-bc556aa4e14b" (UID: "11444a1c-ba90-445f-a682-bc556aa4e14b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:18.573518 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.573460 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-home" (OuterVolumeSpecName: "home") pod "11444a1c-ba90-445f-a682-bc556aa4e14b" (UID: "11444a1c-ba90-445f-a682-bc556aa4e14b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:18.575291 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.575263 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-dshm" (OuterVolumeSpecName: "dshm") pod "11444a1c-ba90-445f-a682-bc556aa4e14b" (UID: "11444a1c-ba90-445f-a682-bc556aa4e14b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:18.575416 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.575305 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11444a1c-ba90-445f-a682-bc556aa4e14b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11444a1c-ba90-445f-a682-bc556aa4e14b" (UID: "11444a1c-ba90-445f-a682-bc556aa4e14b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:18:18.575416 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.575310 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11444a1c-ba90-445f-a682-bc556aa4e14b-kube-api-access-n28kt" (OuterVolumeSpecName: "kube-api-access-n28kt") pod "11444a1c-ba90-445f-a682-bc556aa4e14b" (UID: "11444a1c-ba90-445f-a682-bc556aa4e14b"). InnerVolumeSpecName "kube-api-access-n28kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:18:18.674865 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.674766 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:18.675047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.674909 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:18.675047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.674947 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11444a1c-ba90-445f-a682-bc556aa4e14b-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:18.675047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.674968 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:18.675047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.674976 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28kt\" (UniqueName: \"kubernetes.io/projected/11444a1c-ba90-445f-a682-bc556aa4e14b-kube-api-access-n28kt\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:18.675047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.674985 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11444a1c-ba90-445f-a682-bc556aa4e14b-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:18.904510 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.904471 2562 generic.go:356] "Generic (PLEG): container finished" podID="11444a1c-ba90-445f-a682-bc556aa4e14b" containerID="91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f" exitCode=0 Jul 06 14:18:18.904678 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.904517 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" Jul 06 14:18:18.904678 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.904557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" event={"ID":"11444a1c-ba90-445f-a682-bc556aa4e14b","Type":"ContainerDied","Data":"91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f"} Jul 06 14:18:18.904678 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.904597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n" event={"ID":"11444a1c-ba90-445f-a682-bc556aa4e14b","Type":"ContainerDied","Data":"c40085222773f82bccfb7e7e591ffe2b12f8f2719941ec2c46e9d65b7894472c"} Jul 06 14:18:18.904678 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.904615 2562 scope.go:117] "RemoveContainer" containerID="91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f" Jul 06 14:18:18.913899 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.913876 2562 scope.go:117] "RemoveContainer" containerID="91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f" Jul 06 14:18:18.914167 ip-10-0-134-202 kubenswrapper[2562]: E0706 14:18:18.914144 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f\": container with ID starting with 91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f not found: ID does not exist" containerID="91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f" Jul 06 14:18:18.914227 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.914176 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f"} err="failed to get container status \"91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f\": rpc error: code = NotFound desc = could not find container \"91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f\": container with ID starting with 91c3595b04d51cf2f4ea9a7be22b0f3565a8deb7566542f94118bdf8e975313f not found: ID does not exist" Jul 06 14:18:18.930636 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.930582 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n"] Jul 06 14:18:18.933914 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:18.933893 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-973328/rate-test-model-973328-kserve-84794d466c-r6d6n"] Jul 06 14:18:19.078429 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.078386 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" podUID="f246e29a-393c-44ee-b7f0-f55a183fdf8e" containerName="main" containerID="cri-o://003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461" gracePeriod=45 Jul 06 14:18:19.318961 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.318935 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:18:19.481654 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481557 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tls-certs\") pod \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " Jul 06 14:18:19.481654 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481607 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tmp-dir\") pod \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " Jul 06 14:18:19.481885 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481662 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-home\") pod \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " Jul 06 14:18:19.481885 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481721 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-dshm\") pod \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " Jul 06 14:18:19.481885 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9wmm\" (UniqueName: \"kubernetes.io/projected/f246e29a-393c-44ee-b7f0-f55a183fdf8e-kube-api-access-r9wmm\") pod \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " Jul 06 14:18:19.481885 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481767 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-model-cache\") pod \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\" (UID: \"f246e29a-393c-44ee-b7f0-f55a183fdf8e\") " Jul 06 14:18:19.482102 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481959 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-home" (OuterVolumeSpecName: "home") pod "f246e29a-393c-44ee-b7f0-f55a183fdf8e" (UID: "f246e29a-393c-44ee-b7f0-f55a183fdf8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:19.482102 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.481970 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f246e29a-393c-44ee-b7f0-f55a183fdf8e" (UID: "f246e29a-393c-44ee-b7f0-f55a183fdf8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:19.482215 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.482161 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f246e29a-393c-44ee-b7f0-f55a183fdf8e" (UID: "f246e29a-393c-44ee-b7f0-f55a183fdf8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:19.483962 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.483937 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-dshm" (OuterVolumeSpecName: "dshm") pod "f246e29a-393c-44ee-b7f0-f55a183fdf8e" (UID: "f246e29a-393c-44ee-b7f0-f55a183fdf8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 14:18:19.484083 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.483975 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f246e29a-393c-44ee-b7f0-f55a183fdf8e" (UID: "f246e29a-393c-44ee-b7f0-f55a183fdf8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 14:18:19.484083 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.484015 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f246e29a-393c-44ee-b7f0-f55a183fdf8e-kube-api-access-r9wmm" (OuterVolumeSpecName: "kube-api-access-r9wmm") pod "f246e29a-393c-44ee-b7f0-f55a183fdf8e" (UID: "f246e29a-393c-44ee-b7f0-f55a183fdf8e"). InnerVolumeSpecName "kube-api-access-r9wmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 14:18:19.543714 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.543669 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11444a1c-ba90-445f-a682-bc556aa4e14b" path="/var/lib/kubelet/pods/11444a1c-ba90-445f-a682-bc556aa4e14b/volumes" Jul 06 14:18:19.583238 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.583197 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-dshm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:19.583238 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.583228 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9wmm\" (UniqueName: \"kubernetes.io/projected/f246e29a-393c-44ee-b7f0-f55a183fdf8e-kube-api-access-r9wmm\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:19.583238 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.583239 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-model-cache\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:19.583238 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.583249 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tls-certs\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:19.583506 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.583257 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-tmp-dir\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:19.583506 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.583264 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f246e29a-393c-44ee-b7f0-f55a183fdf8e-home\") on node \"ip-10-0-134-202.ec2.internal\" DevicePath \"\"" Jul 06 14:18:19.912168 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.912133 2562 generic.go:356] "Generic (PLEG): container finished" podID="f246e29a-393c-44ee-b7f0-f55a183fdf8e" containerID="003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461" exitCode=0 Jul 06 14:18:19.912345 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.912181 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" Jul 06 14:18:19.912345 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.912218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" event={"ID":"f246e29a-393c-44ee-b7f0-f55a183fdf8e","Type":"ContainerDied","Data":"003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461"} Jul 06 14:18:19.912345 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.912266 2562 scope.go:117] "RemoveContainer" containerID="003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461" Jul 06 14:18:19.912654 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.912598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2" event={"ID":"f246e29a-393c-44ee-b7f0-f55a183fdf8e","Type":"ContainerDied","Data":"cacfe571e32859473829872854c8777616edcefdafb365af757debc1914789bb"} Jul 06 14:18:19.928049 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.928031 2562 scope.go:117] "RemoveContainer" containerID="003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461" Jul 06 14:18:19.928321 ip-10-0-134-202 kubenswrapper[2562]: E0706 14:18:19.928300 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461\": container with ID starting with 003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461 not found: ID does not exist" containerID="003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461" Jul 06 14:18:19.928396 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.928331 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461"} err="failed to get container status \"003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461\": rpc error: code = NotFound desc = could not find container \"003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461\": container with ID starting with 003c01f4a93cf5f4e7343322a71f50847c274a9312f4a92db7a730d6caacd461 not found: ID does not exist" Jul 06 14:18:19.937810 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.937771 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2"] Jul 06 14:18:19.939817 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:19.939798 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-35c041/rate-test-model-35c041-kserve-667995d5cf-bdpb2"] Jul 06 14:18:21.544190 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:21.544155 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f246e29a-393c-44ee-b7f0-f55a183fdf8e" path="/var/lib/kubelet/pods/f246e29a-393c-44ee-b7f0-f55a183fdf8e/volumes" Jul 06 14:18:48.227309 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:48.227266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:48.492567 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:48.492482 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:48.788713 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:48.788628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:48.919816 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:48.919758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:49.297220 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:49.297186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:49.398004 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:49.397972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:49.498047 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:49.498012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:49.588258 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:49.588175 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:52.300432 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:52.300395 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5wdlq"] Jul 06 14:18:55.338470 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:55.338432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:55.408230 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:55.408198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:55.631869 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:55.631768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:55.865365 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:55.865334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:57.475615 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:57.475566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:18:57.576941 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:18:57.576907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:19:01.494897 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:01.494858 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722455-46xlw_e8ef975d-07c9-48f6-a9da-9a5b1d31858d/cleanup/0.log" Jul 06 14:19:02.255729 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:02.255703 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b559b5dcd-q99dd_06719a70-b17b-4407-bd61-b99e5fc36fce/copy-manifests/0.log" Jul 06 14:19:02.271900 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:02.271871 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b559b5dcd-q99dd_06719a70-b17b-4407-bd61-b99e5fc36fce/manager/0.log" Jul 06 14:19:03.181608 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.181575 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb_e6052d41-3499-4aba-a87e-e7dcf7c62102/pull/0.log" Jul 06 14:19:03.188390 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.188367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb_e6052d41-3499-4aba-a87e-e7dcf7c62102/extract/0.log" Jul 06 14:19:03.195146 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.195126 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb_e6052d41-3499-4aba-a87e-e7dcf7c62102/util/0.log" Jul 06 14:19:03.308392 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.308359 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979_91542230-b669-4172-9839-de962f36d88f/util/0.log" Jul 06 14:19:03.317373 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.317347 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979_91542230-b669-4172-9839-de962f36d88f/pull/0.log" Jul 06 14:19:03.323768 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.323750 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979_91542230-b669-4172-9839-de962f36d88f/extract/0.log" Jul 06 14:19:03.450211 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.450131 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc_6fc842d6-81b4-4f45-801c-9a374e1252c2/pull/0.log" Jul 06 14:19:03.456621 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.456597 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc_6fc842d6-81b4-4f45-801c-9a374e1252c2/extract/0.log" Jul 06 14:19:03.462461 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.462438 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc_6fc842d6-81b4-4f45-801c-9a374e1252c2/util/0.log" Jul 06 14:19:03.572601 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.572574 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h_8060ff05-57ac-42d7-b849-f81c671fb0b0/util/0.log" Jul 06 14:19:03.580633 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.580616 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h_8060ff05-57ac-42d7-b849-f81c671fb0b0/pull/0.log" Jul 06 14:19:03.587044 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.587021 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h_8060ff05-57ac-42d7-b849-f81c671fb0b0/extract/0.log" Jul 06 14:19:03.954125 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:03.954093 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wpcd8_7ce4c2e8-953e-4446-9c7c-2a5015cf6273/manager/0.log" Jul 06 14:19:04.060771 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:04.060734 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hgw2v_92681fff-cf12-42ad-b7bf-7ae7936e2691/kuadrant-console-plugin/0.log" Jul 06 14:19:04.411868 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:04.411840 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5wdlq_a4ceec97-7568-4765-ac03-c8a388c17950/limitador/0.log" Jul 06 14:19:05.241331 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:05.241302 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bfc8ccc9-pj6cx_c03f7f14-040a-43f9-8372-27db7900efb4/kube-auth-proxy/0.log" Jul 06 14:19:06.407587 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:06.407557 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbpsxrh_60425df0-fb04-4d4b-92bf-ede71cdf72b9/main/0.log" Jul 06 14:19:06.640206 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:06.640162 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-bgrlz_7bdcfd45-1368-4bb4-8853-7e9f34e7d694/main/0.log" Jul 06 14:19:10.483745 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:10.483712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:19:10.637309 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:10.637271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" limit=1024 Jul 06 14:19:13.769364 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:13.769330 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8nr9n_8a3114dd-57c2-4a17-ac11-334b98b10191/global-pull-secret-syncer/0.log" Jul 06 14:19:13.906562 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:13.906535 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5g528_00bf2769-4a95-4dc6-b1c5-825808185157/konnectivity-agent/0.log" Jul 06 14:19:13.996705 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:13.996671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-202.ec2.internal_4b12aea9bf8b99530f8679e8c6d1d16c/haproxy/0.log" Jul 06 14:19:18.483264 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.483233 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb_e6052d41-3499-4aba-a87e-e7dcf7c62102/extract/0.log" Jul 06 14:19:18.519772 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.519746 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb_e6052d41-3499-4aba-a87e-e7dcf7c62102/util/0.log" Jul 06 14:19:18.544619 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.544590 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595xwzb_e6052d41-3499-4aba-a87e-e7dcf7c62102/pull/0.log" Jul 06 14:19:18.581808 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.581768 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979_91542230-b669-4172-9839-de962f36d88f/extract/0.log" Jul 06 14:19:18.608944 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.608924 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979_91542230-b669-4172-9839-de962f36d88f/util/0.log" Jul 06 14:19:18.636092 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.636067 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08j979_91542230-b669-4172-9839-de962f36d88f/pull/0.log" Jul 06 14:19:18.675997 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.675974 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc_6fc842d6-81b4-4f45-801c-9a374e1252c2/extract/0.log" Jul 06 14:19:18.712599 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.712575 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc_6fc842d6-81b4-4f45-801c-9a374e1252c2/util/0.log" Jul 06 14:19:18.737266 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.737215 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73868mc_6fc842d6-81b4-4f45-801c-9a374e1252c2/pull/0.log" Jul 06 14:19:18.764982 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.764956 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h_8060ff05-57ac-42d7-b849-f81c671fb0b0/extract/0.log" Jul 06 14:19:18.788899 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.788873 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h_8060ff05-57ac-42d7-b849-f81c671fb0b0/util/0.log" Jul 06 14:19:18.817367 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:18.817343 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bmk7h_8060ff05-57ac-42d7-b849-f81c671fb0b0/pull/0.log" Jul 06 14:19:19.217227 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:19.217200 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wpcd8_7ce4c2e8-953e-4446-9c7c-2a5015cf6273/manager/0.log" Jul 06 14:19:19.250063 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:19.250035 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hgw2v_92681fff-cf12-42ad-b7bf-7ae7936e2691/kuadrant-console-plugin/0.log" Jul 06 14:19:19.474104 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:19.474019 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5wdlq_a4ceec97-7568-4765-ac03-c8a388c17950/limitador/0.log" Jul 06 14:19:21.390527 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.390501 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/alertmanager/0.log" Jul 06 14:19:21.417738 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.417711 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/config-reloader/0.log" Jul 06 14:19:21.442114 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.442081 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/kube-rbac-proxy-web/0.log" Jul 06 14:19:21.470522 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.470497 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/kube-rbac-proxy/0.log" Jul 06 14:19:21.498191 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.498156 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/kube-rbac-proxy-metric/0.log" Jul 06 14:19:21.538690 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.538663 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/prom-label-proxy/0.log" Jul 06 14:19:21.570240 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.570168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_095d13b6-fbd4-44f4-98f5-420fd9a17caa/init-config-reloader/0.log" Jul 06 14:19:21.679620 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.679571 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mxl7l_fe7f0209-ce35-4970-9545-16905eded10f/kube-state-metrics/0.log" Jul 06 14:19:21.700998 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.700972 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mxl7l_fe7f0209-ce35-4970-9545-16905eded10f/kube-rbac-proxy-main/0.log" Jul 06 14:19:21.726600 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:21.726576 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-mxl7l_fe7f0209-ce35-4970-9545-16905eded10f/kube-rbac-proxy-self/0.log" Jul 06 14:19:22.001236 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.001162 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sx2h2_660e9036-f3f6-4056-8c58-b23e624a5bc7/node-exporter/0.log" Jul 06 14:19:22.029528 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.029502 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sx2h2_660e9036-f3f6-4056-8c58-b23e624a5bc7/kube-rbac-proxy/0.log" Jul 06 14:19:22.055119 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.055099 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sx2h2_660e9036-f3f6-4056-8c58-b23e624a5bc7/init-textfile/0.log" Jul 06 14:19:22.085612 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.085590 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-llfw7_0904070f-36bf-4757-951a-35740c0d8c3f/kube-rbac-proxy-main/0.log" Jul 06 14:19:22.109670 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.109648 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-llfw7_0904070f-36bf-4757-951a-35740c0d8c3f/kube-rbac-proxy-self/0.log" Jul 06 14:19:22.132352 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.132329 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-llfw7_0904070f-36bf-4757-951a-35740c0d8c3f/openshift-state-metrics/0.log" Jul 06 14:19:22.398259 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.398232 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-cl56h_b4d3583f-1deb-4d0d-8406-e9d5f28d79e3/prometheus-operator-admission-webhook/0.log" Jul 06 14:19:22.432898 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.432872 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d6764b494-v5n8v_88ada1f9-9094-4535-ab8d-c210fc19b106/telemeter-client/0.log" Jul 06 14:19:22.455132 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.455111 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d6764b494-v5n8v_88ada1f9-9094-4535-ab8d-c210fc19b106/reload/0.log" Jul 06 14:19:22.477854 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:22.477833 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d6764b494-v5n8v_88ada1f9-9094-4535-ab8d-c210fc19b106/kube-rbac-proxy/0.log" Jul 06 14:19:24.364867 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.364822 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx"] Jul 06 14:19:24.365456 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.365369 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11444a1c-ba90-445f-a682-bc556aa4e14b" containerName="main" Jul 06 14:19:24.365456 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.365389 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11444a1c-ba90-445f-a682-bc556aa4e14b" containerName="main" Jul 06 14:19:24.365456 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.365410 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f246e29a-393c-44ee-b7f0-f55a183fdf8e" containerName="main" Jul 06 14:19:24.365456 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.365418 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f246e29a-393c-44ee-b7f0-f55a183fdf8e" containerName="main" Jul 06 14:19:24.365668 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.365509 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11444a1c-ba90-445f-a682-bc556aa4e14b" containerName="main" Jul 06 14:19:24.365668 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.365521 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f246e29a-393c-44ee-b7f0-f55a183fdf8e" containerName="main" Jul 06 14:19:24.368543 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.368523 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.371157 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.371141 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kwcr\"/\"kube-root-ca.crt\"" Jul 06 14:19:24.371157 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.371149 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kwcr\"/\"openshift-service-ca.crt\"" Jul 06 14:19:24.372276 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.372260 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8kwcr\"/\"default-dockercfg-tv44x\"" Jul 06 14:19:24.376083 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.376062 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx"] Jul 06 14:19:24.533179 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.533141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-podres\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.533356 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.533207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-lib-modules\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.533356 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.533267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-proc\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.533356 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.533329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-sys\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.533489 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.533395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbl6\" (UniqueName: \"kubernetes.io/projected/42594633-eb66-4299-ad62-30e4eba8a491-kube-api-access-gpbl6\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634309 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-lib-modules\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634309 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-proc\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-sys\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634383 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-lib-modules\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634422 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-proc\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634423 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-sys\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbl6\" (UniqueName: \"kubernetes.io/projected/42594633-eb66-4299-ad62-30e4eba8a491-kube-api-access-gpbl6\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634539 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-podres\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.634813 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.634655 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42594633-eb66-4299-ad62-30e4eba8a491-podres\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.642045 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.642017 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbl6\" (UniqueName: \"kubernetes.io/projected/42594633-eb66-4299-ad62-30e4eba8a491-kube-api-access-gpbl6\") pod \"perf-node-gather-daemonset-8cqmx\" (UID: \"42594633-eb66-4299-ad62-30e4eba8a491\") " pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.679240 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.679201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:24.808245 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:24.808213 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx"] Jul 06 14:19:24.809504 ip-10-0-134-202 kubenswrapper[2562]: W0706 14:19:24.809477 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod42594633_eb66_4299_ad62_30e4eba8a491.slice/crio-9fea9695957628c95c8e4f42a4e7c9a5756fea079aed610d14486d56e6723115 WatchSource:0}: Error finding container 9fea9695957628c95c8e4f42a4e7c9a5756fea079aed610d14486d56e6723115: Status 404 returned error can't find the container with id 9fea9695957628c95c8e4f42a4e7c9a5756fea079aed610d14486d56e6723115 Jul 06 14:19:25.188698 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:25.188671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56d476c456-ptdtq_22613d14-3bf2-4c42-97bb-9dc38d8e28db/console/0.log" Jul 06 14:19:25.267325 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:25.267288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" event={"ID":"42594633-eb66-4299-ad62-30e4eba8a491","Type":"ContainerStarted","Data":"c5ef1ce947f022ec476aa57e41c7825b244344a6b92d24c4b89d80ca412ecd4b"} Jul 06 14:19:25.267572 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:25.267346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" event={"ID":"42594633-eb66-4299-ad62-30e4eba8a491","Type":"ContainerStarted","Data":"9fea9695957628c95c8e4f42a4e7c9a5756fea079aed610d14486d56e6723115"} Jul 06 14:19:25.267572 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:25.267468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:25.284931 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:25.284884 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" podStartSLOduration=1.2848679920000001 podStartE2EDuration="1.284867992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 14:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 14:19:25.283851561 +0000 UTC m=+2381.033665148" watchObservedRunningTime="2026-07-06 14:19:25.284867992 +0000 UTC m=+2381.034681557" Jul 06 14:19:25.870968 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:25.870934 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-m5btr_6b0747ae-f0a5-49e5-948d-87fd0edf0529/volume-data-source-validator/0.log" Jul 06 14:19:27.014927 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:27.014900 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffb96_df97a078-18f6-4536-bdc5-c5ea3779faaf/dns/0.log" Jul 06 14:19:27.045074 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:27.045043 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffb96_df97a078-18f6-4536-bdc5-c5ea3779faaf/kube-rbac-proxy/0.log" Jul 06 14:19:27.177068 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:27.177042 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpc42_883b955a-f5b4-4cae-87e5-55ea75c2be59/dns-node-resolver/0.log" Jul 06 14:19:27.762745 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:27.762719 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ptzd_fe5c8515-7f08-4442-ba21-c438df4a8849/node-ca/0.log" Jul 06 14:19:29.069585 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:29.069559 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bfc8ccc9-pj6cx_c03f7f14-040a-43f9-8372-27db7900efb4/kube-auth-proxy/0.log" Jul 06 14:19:30.116132 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:30.116106 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d4mwv_486b80ea-65a7-4727-bed6-ee1437cb6d36/serve-healthcheck-canary/0.log" Jul 06 14:19:30.731497 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:30.731469 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4n8s_6cb10152-577e-4255-ab0b-5539b0b6f8c4/kube-rbac-proxy/0.log" Jul 06 14:19:30.755624 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:30.755593 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4n8s_6cb10152-577e-4255-ab0b-5539b0b6f8c4/exporter/0.log" Jul 06 14:19:30.778333 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:30.778308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4n8s_6cb10152-577e-4255-ab0b-5539b0b6f8c4/extractor/0.log" Jul 06 14:19:31.282702 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:31.282670 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8kwcr/perf-node-gather-daemonset-8cqmx" Jul 06 14:19:33.682650 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:33.682571 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722455-46xlw_e8ef975d-07c9-48f6-a9da-9a5b1d31858d/cleanup/0.log" Jul 06 14:19:34.097327 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:34.097285 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b559b5dcd-q99dd_06719a70-b17b-4407-bd61-b99e5fc36fce/manager/0.log" Jul 06 14:19:34.116109 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:34.116085 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b559b5dcd-q99dd_06719a70-b17b-4407-bd61-b99e5fc36fce/copy-manifests/0.log" Jul 06 14:19:35.553901 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:35.553869 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bfc8ccc9-pj6cx_c03f7f14-040a-43f9-8372-27db7900efb4/kube-auth-proxy/0.log" Jul 06 14:19:35.670429 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:35.670393 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6ff5df8b5c-zr6mh_2e2bf2a8-2b4d-48b0-8cc6-329c4da05acf/manager/0.log" Jul 06 14:19:35.698680 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:35.698638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-946b8_21b6b12a-cbcb-41d4-83e9-c50ce708c331/openshift-lws-operator/0.log" Jul 06 14:19:37.079696 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:37.079671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bfc8ccc9-pj6cx_c03f7f14-040a-43f9-8372-27db7900efb4/kube-auth-proxy/0.log" Jul 06 14:19:41.134241 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:41.134212 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-5qddz_41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93/migrator/0.log" Jul 06 14:19:41.157982 ip-10-0-134-202 kubenswrapper[2562]: I0706 14:19:41.157960 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-5qddz_41d31968-0fa2-47f0-b7e2-7f2f7d2e0b93/graceful-termination/0.log"