Jul 28 17:07:12.689208 ip-10-0-134-110 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 17:07:12.689221 ip-10-0-134-110 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 17:07:12.689285 ip-10-0-134-110 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 17:07:12.689540 ip-10-0-134-110 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 17:07:22.820707 ip-10-0-134-110 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 17:07:22.820725 ip-10-0-134-110 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f49b3fface7f4e9285d439f7097db916 -- Jul 28 17:09:17.841168 ip-10-0-134-110 systemd[1]: Starting Kubernetes Kubelet... Jul 28 17:09:18.142662 ip-10-0-134-110 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:09:18.142662 ip-10-0-134-110 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 17:09:18.142662 ip-10-0-134-110 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:09:18.142662 ip-10-0-134-110 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 17:09:18.142662 ip-10-0-134-110 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:09:18.144107 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.144019 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 17:09:18.150105 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150089 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:09:18.150105 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150105 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150110 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150113 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150116 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150119 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150122 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150124 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150127 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150130 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150137 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150140 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150143 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150145 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150148 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150151 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150153 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150158 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150161 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150164 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:09:18.150171 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150167 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150171 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150175 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150178 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150182 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150185 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150187 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150190 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150193 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150197 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150200 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150203 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150206 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150209 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150211 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150214 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150216 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150219 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150221 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:09:18.150626 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150224 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150226 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150229 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150232 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150235 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150237 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150239 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150242 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150244 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150247 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150249 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150252 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150255 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150257 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150260 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150263 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150266 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150269 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150271 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150274 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:09:18.151143 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150276 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150279 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150282 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150285 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150287 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150291 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150293 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150297 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150299 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150302 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150304 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150307 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150310 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150313 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150315 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150318 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150320 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150323 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150325 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150328 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:09:18.151641 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150330 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150333 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150335 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150338 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150340 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150343 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150346 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150349 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150352 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150355 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150357 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150360 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150362 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150364 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150367 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150370 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150372 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150375 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150377 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.150380 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:09:18.152149 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151106 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151113 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151116 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151119 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151121 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151124 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151128 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151130 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151138 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151141 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151144 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151147 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151150 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151153 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151155 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151158 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151161 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151164 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151166 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151170 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:09:18.152649 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151173 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151175 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151178 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151180 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151183 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151187 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151191 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151194 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151197 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151201 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151204 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151206 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151210 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151213 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151218 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151222 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151226 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151229 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151232 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:09:18.153164 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151235 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151238 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151248 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151251 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151255 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151257 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151260 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151263 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151265 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151268 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151272 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151282 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151286 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151288 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151291 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151294 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151296 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151299 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151301 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151304 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:09:18.153659 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151306 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151309 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151311 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151314 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151316 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151319 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151321 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151324 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151326 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151328 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151331 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151333 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151336 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151338 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151341 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151349 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151352 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151354 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151357 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151359 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:09:18.154173 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151362 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151365 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151368 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151371 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151374 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151376 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151379 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151381 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151384 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151386 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151389 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151391 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151394 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151396 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151398 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151401 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151403 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151406 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151408 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151411 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:09:18.154675 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151781 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151786 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151789 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151792 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151794 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151797 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151800 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151809 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151812 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151816 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151821 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151824 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151827 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151830 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151833 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151836 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151839 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151842 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151846 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:09:18.155186 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151848 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151851 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151854 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151856 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151871 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151874 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151877 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151879 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151882 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151885 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151887 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151890 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151893 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151895 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151898 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151901 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151906 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151908 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151911 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151914 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:09:18.155663 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151917 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151925 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151928 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151931 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151933 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151936 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151939 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151941 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151944 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151946 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151949 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151951 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151954 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151956 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151959 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151961 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151964 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151966 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151969 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151971 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:09:18.156194 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151974 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151978 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151981 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151983 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151986 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151988 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151991 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151993 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.151997 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152000 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152003 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152007 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152010 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152012 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152022 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152025 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152028 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152030 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152033 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152036 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:09:18.156679 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152038 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152041 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152044 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152047 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152049 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152051 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152054 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152056 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152059 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152061 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152064 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152067 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152069 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152072 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152074 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152077 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152079 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152082 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152090 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152092 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:09:18.157183 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152170 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152176 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152182 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152190 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152197 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152201 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152207 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152220 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152224 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152227 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152230 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152234 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152237 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152240 2567 flags.go:64] FLAG: --cgroup-root="" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152243 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152246 2567 flags.go:64] FLAG: --client-ca-file="" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152249 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152252 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152261 2567 flags.go:64] FLAG: --cluster-domain="" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152264 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152267 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152270 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152273 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152277 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 17:09:18.157680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152280 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152283 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152286 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152290 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152293 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152296 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152299 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152302 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152307 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152310 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152313 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152318 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152321 2567 flags.go:64] FLAG: --enable-server="true" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152324 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152330 2567 flags.go:64] FLAG: --event-burst="100" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152333 2567 flags.go:64] FLAG: --event-qps="50" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152341 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152345 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152348 2567 flags.go:64] FLAG: --eviction-hard="" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152351 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152355 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152358 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152361 2567 flags.go:64] FLAG: --eviction-soft="" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152363 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152367 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 17:09:18.158289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152370 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152372 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152375 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152378 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152381 2567 flags.go:64] FLAG: --feature-gates="" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152387 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152390 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152393 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152396 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152399 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152402 2567 flags.go:64] FLAG: --help="false" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152405 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152408 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152411 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152414 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152419 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152422 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152426 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152429 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152432 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152435 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152438 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152441 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152444 2567 flags.go:64] FLAG: --kube-reserved="" Jul 28 17:09:18.158989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152454 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152457 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152460 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152463 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152466 2567 flags.go:64] FLAG: --lock-file="" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152468 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152471 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152474 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152480 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152483 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152485 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152488 2567 flags.go:64] FLAG: --logging-format="text" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152491 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152494 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152497 2567 flags.go:64] FLAG: --manifest-url="" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152500 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152504 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152506 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152510 2567 flags.go:64] FLAG: --max-pods="110" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152513 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152516 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152519 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152521 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152526 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152529 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 17:09:18.159590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152532 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152540 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152543 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152546 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152549 2567 flags.go:64] FLAG: --pod-cidr="" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152552 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152561 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152564 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152573 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152576 2567 flags.go:64] FLAG: --port="10250" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152579 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152581 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a458283e86eb7b43" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152585 2567 flags.go:64] FLAG: --qos-reserved="" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152588 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152590 2567 flags.go:64] FLAG: --register-node="true" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152593 2567 flags.go:64] FLAG: --register-with-taints="" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152601 2567 flags.go:64] FLAG: --registry-burst="10" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152603 2567 flags.go:64] FLAG: --registry-qps="5" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152606 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152609 2567 flags.go:64] FLAG: --reserved-memory="" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152613 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152616 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152619 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152621 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152624 2567 flags.go:64] FLAG: --runonce="false" Jul 28 17:09:18.160335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152627 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152630 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152633 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152636 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152638 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152641 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152646 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152649 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152653 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152656 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152659 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152662 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152665 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152668 2567 flags.go:64] FLAG: --system-cgroups="" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152671 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152676 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152685 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152688 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152700 2567 flags.go:64] FLAG: --tls-min-version="" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152703 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152706 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152708 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152711 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152714 2567 flags.go:64] FLAG: --v="2" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152718 2567 flags.go:64] FLAG: --version="false" Jul 28 17:09:18.161316 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152721 2567 flags.go:64] FLAG: --vmodule="" Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152725 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.152728 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152916 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152922 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152925 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152928 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152931 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152934 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152936 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152939 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152941 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152945 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152949 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152951 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152955 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152957 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152960 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152963 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152965 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:09:18.161971 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152968 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152970 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152973 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152975 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152985 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152988 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152991 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152993 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152996 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.152998 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153001 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153004 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153006 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153008 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153011 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153015 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153018 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153021 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153024 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:09:18.162483 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153027 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153030 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153032 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153035 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153037 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153040 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153044 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153047 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153049 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153052 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153054 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153057 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153060 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153062 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153065 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153067 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153073 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153075 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153084 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153086 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:09:18.162978 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153090 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153094 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153097 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153100 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153104 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153106 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153109 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153112 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153114 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153117 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153119 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153122 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153124 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153127 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153129 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153132 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153134 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153137 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153141 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:09:18.163506 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153143 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153146 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153148 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153151 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153154 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153156 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153159 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153161 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153164 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153166 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153169 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153172 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153180 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153183 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153186 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153188 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153191 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153193 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153196 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153198 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:09:18.164013 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153201 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153203 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153206 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.153208 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.153213 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.161840 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.161855 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.162515 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.162527 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162585 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162590 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162594 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162597 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162600 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162603 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:09:18.164492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162606 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162610 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162614 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162617 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162620 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162623 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162626 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162629 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162632 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162635 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162638 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162641 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162643 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162646 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162649 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162653 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162656 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162659 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162662 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:09:18.164908 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162665 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162668 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162671 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162674 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162677 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162680 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162683 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162686 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162689 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162691 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162694 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162697 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162699 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162702 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162704 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162707 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162709 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162712 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162715 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162717 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:09:18.165370 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162720 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162722 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162725 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162728 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162730 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162733 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162735 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162738 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162740 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162743 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162745 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162748 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162750 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162753 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162755 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162758 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162761 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162764 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162767 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162769 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:09:18.165924 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162772 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162774 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162777 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162779 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162782 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162784 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162787 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162789 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162791 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162794 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162797 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162799 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162802 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162804 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162814 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162818 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162821 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162823 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162826 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162829 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:09:18.166432 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162831 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162834 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162836 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162839 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162842 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162844 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162847 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162850 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162853 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162856 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162858 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162875 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162879 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.162882 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.162887 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:09:18.167081 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163014 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163020 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163024 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163027 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163030 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163033 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163036 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163038 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163041 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163044 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163047 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163049 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163059 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163062 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163064 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163067 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163069 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163072 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163074 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163077 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:09:18.167472 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163079 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163082 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163084 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163087 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163090 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163093 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163096 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163098 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163101 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163103 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163106 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163108 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163111 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163113 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163116 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163118 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163120 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163124 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163126 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163129 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:09:18.168058 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163131 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163135 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163139 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163143 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163146 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163155 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163158 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163160 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163163 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163165 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163168 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163171 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163173 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163177 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163180 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163183 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163186 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163189 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:09:18.168782 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163192 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163194 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163197 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163199 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163202 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163204 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163207 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163210 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163212 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163215 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163217 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163220 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163222 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163225 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163227 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163230 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163232 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163235 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163237 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:09:18.169249 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163240 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163243 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163246 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163248 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163251 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163253 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163256 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163258 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163261 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163263 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163265 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163268 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163271 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163279 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163283 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163285 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163288 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163291 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163293 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163296 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:09:18.169742 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163298 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:18.163301 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.163306 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.163408 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.165500 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.166670 2567 server.go:1019] "Starting client certificate rotation" Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.166781 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.166832 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.168627 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:09:18.170352 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.169802 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:09:18.176094 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.176078 2567 log.go:25] "Validated CRI v1 runtime API" Jul 28 17:09:18.178122 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.178109 2567 log.go:25] "Validated CRI v1 image API" Jul 28 17:09:18.178900 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.178885 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 17:09:18.180845 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.180829 2567 fs.go:135] Filesystem UUIDs: map[02bd6d88-3b9a-4413-8cc3-ee812d1d7855:/dev/nvme0n1p3 3e580eff-4c4d-4afc-b6df-972ee5b65dff:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 28 17:09:18.180905 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.180845 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 17:09:18.186586 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.186470 2567 manager.go:216] Machine: {Timestamp:2026-07-28 17:09:18.185329327 +0000 UTC m=+0.250825284 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099115 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287caf7e13321f4d128af90a9064ab SystemUUID:ec287caf-7e13-321f-4d12-8af90a9064ab BootID:f49b3ffa-ce7f-4e92-85d4-39f7097db916 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:bf:dd:52:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:bf:dd:52:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:e6:fc:8f:17:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 17:09:18.186586 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.186580 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 17:09:18.186707 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.186657 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 17:09:18.188748 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.188721 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 17:09:18.188927 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.188752 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 17:09:18.189412 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.189402 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 17:09:18.189445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.189416 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 17:09:18.189445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.189423 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:09:18.189445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.189436 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:09:18.190185 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.190175 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 17:09:18.193266 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.193252 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:09:18.193387 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.193377 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 17:09:18.194013 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194004 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 28 17:09:18.194046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194017 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 17:09:18.194046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194035 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 17:09:18.194046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194044 2567 kubelet.go:398] "Adding apiserver pod source" Jul 28 17:09:18.194148 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194052 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 17:09:18.194463 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194450 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:09:18.194520 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.194470 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:09:18.196199 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.196180 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 17:09:18.198137 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.198119 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 17:09:18.198209 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.198168 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 17:09:18.199668 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199655 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199671 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199677 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199683 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199692 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199697 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199705 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199710 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 17:09:18.199715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199716 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 17:09:18.199935 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199722 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 17:09:18.199935 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199731 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 17:09:18.199935 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.199736 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 17:09:18.200298 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.200284 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 17:09:18.200298 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.200299 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 17:09:18.202238 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.202216 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:09:18.203332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.203313 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 17:09:18.203419 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.203365 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 17:09:18.203762 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.203747 2567 server.go:1268] "Started kubelet" Jul 28 17:09:18.203832 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.203785 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 17:09:18.203890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.203844 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 17:09:18.204144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.204000 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 17:09:18.204264 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.204252 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 17:09:18.205640 ip-10-0-134-110 systemd[1]: Started Kubernetes Kubelet. Jul 28 17:09:18.205846 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.205708 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 17:09:18.206801 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.206744 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 28 17:09:18.209352 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.209333 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 17:09:18.211913 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.211890 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 17:09:18.212302 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.212283 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 17:09:18.212907 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.212887 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 17:09:18.212907 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.212908 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 17:09:18.213046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.212981 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 17:09:18.213095 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213050 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 28 17:09:18.213095 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213056 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 28 17:09:18.213198 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213181 2567 factory.go:55] Registering systemd factory Jul 28 17:09:18.213240 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213203 2567 factory.go:223] Registration of the systemd container factory successfully Jul 28 17:09:18.213329 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.213300 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.213462 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213447 2567 factory.go:153] Registering CRI-O factory Jul 28 17:09:18.213462 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213461 2567 factory.go:223] Registration of the crio container factory successfully Jul 28 17:09:18.213594 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213510 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 17:09:18.213594 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213546 2567 factory.go:103] Registering Raw factory Jul 28 17:09:18.213594 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.213560 2567 manager.go:1196] Started watching for new ooms in manager Jul 28 17:09:18.213777 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.212939 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-110.ec2.internal.18c6833331f59521 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-110.ec2.internal,UID:ip-10-0-134-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-110.ec2.internal,},FirstTimestamp:2026-07-28 17:09:18.203729185 +0000 UTC m=+0.269225142,LastTimestamp:2026-07-28 17:09:18.203729185 +0000 UTC m=+0.269225142,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-110.ec2.internal,}" Jul 28 17:09:18.214443 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.214429 2567 manager.go:349] Starting recovery of all containers Jul 28 17:09:18.215042 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.215004 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 17:09:18.227405 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.227367 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 17:09:18.227520 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.227503 2567 manager.go:354] Recovery completed Jul 28 17:09:18.229125 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.229109 2567 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 28 17:09:18.232003 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.231992 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.233882 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.233844 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.233941 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.233896 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.233941 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.233907 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.234470 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.234454 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 17:09:18.234470 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.234470 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 17:09:18.234470 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.234457 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.234626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.234488 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:09:18.235129 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.235114 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.235199 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.235138 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.235199 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.235148 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.236330 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.236315 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h6rvp" Jul 28 17:09:18.237056 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.237043 2567 policy_none.go:49] "None policy: Start" Jul 28 17:09:18.237115 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.237063 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 17:09:18.237115 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.237104 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 17:09:18.238980 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.238967 2567 policy_none.go:47] "Start" Jul 28 17:09:18.244235 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.244170 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-110.ec2.internal.18c6833333c1a62d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-110.ec2.internal,UID:ip-10-0-134-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-110.ec2.internal,},FirstTimestamp:2026-07-28 17:09:18.233880109 +0000 UTC m=+0.299376073,LastTimestamp:2026-07-28 17:09:18.233880109 +0000 UTC m=+0.299376073,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-110.ec2.internal,}" Jul 28 17:09:18.249089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.249074 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h6rvp" Jul 28 17:09:18.276567 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.276409 2567 manager.go:337] "Starting Device Plugin manager" Jul 28 17:09:18.276656 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.276588 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 17:09:18.276656 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.276601 2567 server.go:87] "Starting device plugin registration server" Jul 28 17:09:18.276892 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.276880 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 17:09:18.276929 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.276898 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 17:09:18.277011 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.276990 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 17:09:18.277109 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.277080 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 17:09:18.277109 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.277089 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 17:09:18.277615 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.277594 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 17:09:18.277708 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.277642 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.377043 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.377001 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.379105 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.378062 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.379105 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.378093 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.379105 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.378103 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.379105 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.378127 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.386260 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.386237 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.386370 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.386265 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-110.ec2.internal\": node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.400683 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.400636 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.413407 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.413377 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 17:09:18.414591 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.414576 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 17:09:18.414649 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.414606 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 17:09:18.414649 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.414624 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 17:09:18.414752 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.414656 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 17:09:18.417410 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.417391 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:09:18.501701 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.501665 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.514918 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.514883 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal"] Jul 28 17:09:18.515008 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.514963 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.516418 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.516401 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.516528 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.516430 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.516528 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.516440 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.518735 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.518722 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.518835 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.518822 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.518890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.518852 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.519411 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.519391 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.519524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.519395 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.519524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.519447 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.519524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.519460 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.519524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.519421 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.519524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.519497 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.521658 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.521644 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.521723 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.521674 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:18.522343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.522328 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:18.522430 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.522356 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:18.522430 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.522369 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:18.535391 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.535373 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-110.ec2.internal\" not found" node="ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.539251 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.539237 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-110.ec2.internal\" not found" node="ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.602717 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.602694 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.703128 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.703050 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.714396 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.714373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/522862f2528540f980092f7e64425196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal\" (UID: \"522862f2528540f980092f7e64425196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.714499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.714404 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9005259af348bbf3d574131a972e70c3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-110.ec2.internal\" (UID: \"9005259af348bbf3d574131a972e70c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.714499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.714433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/522862f2528540f980092f7e64425196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal\" (UID: \"522862f2528540f980092f7e64425196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.804842 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.804801 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:18.814803 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.814783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9005259af348bbf3d574131a972e70c3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-110.ec2.internal\" (UID: \"9005259af348bbf3d574131a972e70c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.814901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.814809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/522862f2528540f980092f7e64425196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal\" (UID: \"522862f2528540f980092f7e64425196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.814901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.814830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/522862f2528540f980092f7e64425196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal\" (UID: \"522862f2528540f980092f7e64425196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.814901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.814858 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/522862f2528540f980092f7e64425196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal\" (UID: \"522862f2528540f980092f7e64425196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.815044 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.814920 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/522862f2528540f980092f7e64425196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal\" (UID: \"522862f2528540f980092f7e64425196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.815044 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.814921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9005259af348bbf3d574131a972e70c3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-110.ec2.internal\" (UID: \"9005259af348bbf3d574131a972e70c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.837656 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.837625 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.843274 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:18.843093 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" Jul 28 17:09:18.906702 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:18.906675 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.013975 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:19.013902 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.116150 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:19.116124 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.122448 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.122421 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:09:19.166560 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.166532 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 17:09:19.167007 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.166688 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:09:19.167007 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.166703 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:09:19.217270 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:19.217235 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.253853 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.253815 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:04:18 +0000 UTC" deadline="2027-12-22 08:29:13.66050386 +0000 UTC" Jul 28 17:09:19.253853 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.253844 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12279h19m54.406663752s" Jul 28 17:09:19.317363 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:19.317333 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.360699 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.360665 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:09:19.361966 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.361947 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:09:19.362076 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.361981 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:09:19.362076 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.361992 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:09:19.418366 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:19.418341 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.427257 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:19.427226 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522862f2528540f980092f7e64425196.slice/crio-aaa514cb7e5c5005b0c361258dea9b23ee56794ce997f98f1daa9efba8b2ff43 WatchSource:0}: Error finding container aaa514cb7e5c5005b0c361258dea9b23ee56794ce997f98f1daa9efba8b2ff43: Status 404 returned error can't find the container with id aaa514cb7e5c5005b0c361258dea9b23ee56794ce997f98f1daa9efba8b2ff43 Jul 28 17:09:19.427652 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:19.427627 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9005259af348bbf3d574131a972e70c3.slice/crio-8a3fda707e748c126086e47b05d17c70bfce383776d289c85096dd0c50394e2a WatchSource:0}: Error finding container 8a3fda707e748c126086e47b05d17c70bfce383776d289c85096dd0c50394e2a: Status 404 returned error can't find the container with id 8a3fda707e748c126086e47b05d17c70bfce383776d289c85096dd0c50394e2a Jul 28 17:09:19.432717 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.432698 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:09:19.433989 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.433974 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:09:19.518800 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:19.518763 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-110.ec2.internal\" not found" Jul 28 17:09:19.578969 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.578901 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:09:19.614599 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.614565 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" Jul 28 17:09:19.631176 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.631152 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:09:19.632052 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.632037 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" Jul 28 17:09:19.641416 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:19.641397 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:09:20.065272 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.065189 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:09:20.196937 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.196907 2567 apiserver.go:52] "Watching apiserver" Jul 28 17:09:20.207443 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.207414 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 17:09:20.208460 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.208430 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hfc7r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-44qsl","openshift-network-diagnostics/network-check-target-2bfzx","openshift-network-operator/iptables-alerter-jx24d","kube-system/global-pull-secret-syncer-2s7mz","kube-system/konnectivity-agent-n4s7j","kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal","openshift-image-registry/node-ca-z5g7t","openshift-multus/multus-46d7c","openshift-multus/network-metrics-daemon-nlxkb","openshift-ovn-kubernetes/ovnkube-node-swcbs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs","openshift-cluster-node-tuning-operator/tuned-2tf6f"] Jul 28 17:09:20.211178 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.211155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.212566 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.212539 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 17:09:20.213649 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.213623 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.214153 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.214123 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 17:09:20.214265 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.214179 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-24vm7\"" Jul 28 17:09:20.214265 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.214222 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 17:09:20.216215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.216188 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2mmw2\"" Jul 28 17:09:20.216337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.216324 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 17:09:20.216401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.216354 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.216516 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.216498 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.216615 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.216598 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.218870 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.218664 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:20.218870 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.218735 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:20.219234 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.219214 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2v4dj\"" Jul 28 17:09:20.219584 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.219560 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 17:09:20.219669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.219564 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 17:09:20.219669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.219568 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.220050 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.220030 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.222105 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.221908 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.223849 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.223819 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mg2\" (UniqueName: \"kubernetes.io/projected/00143af9-f792-4786-9bd2-aa61b5d1c32a-kube-api-access-78mg2\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.223966 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.223868 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c7c6e25-8447-40b7-b2b1-52d9942add01-agent-certs\") pod \"konnectivity-agent-n4s7j\" (UID: \"6c7c6e25-8447-40b7-b2b1-52d9942add01\") " pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.223966 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.223909 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c7c6e25-8447-40b7-b2b1-52d9942add01-konnectivity-ca\") pod \"konnectivity-agent-n4s7j\" (UID: \"6c7c6e25-8447-40b7-b2b1-52d9942add01\") " pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.223966 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.223924 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:09:20.223966 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.223933 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00143af9-f792-4786-9bd2-aa61b5d1c32a-host\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.223966 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.223958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00143af9-f792-4786-9bd2-aa61b5d1c32a-serviceca\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.224524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.224503 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.224603 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.224505 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 17:09:20.224783 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.224763 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.224885 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.224786 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p7nl2\"" Jul 28 17:09:20.225050 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.225032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.225157 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.225134 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:20.227033 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.227011 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.229791 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.229485 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.230517 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.229981 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.230517 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.230018 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 17:09:20.231284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.230953 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-49g7g\"" Jul 28 17:09:20.231284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.231072 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.232194 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.232175 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.232719 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.232699 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z5p7b\"" Jul 28 17:09:20.233095 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.233072 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.233231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.233208 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.236424 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.236236 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 17:09:20.236597 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.236563 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7kdf\"" Jul 28 17:09:20.236689 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.236658 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 17:09:20.237208 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.237187 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:20.237311 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.237260 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:20.239657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.239633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.242283 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.242261 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.243113 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.242921 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 17:09:20.244321 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.244299 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mlhjz\"" Jul 28 17:09:20.244523 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.244506 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 17:09:20.244710 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.244695 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 17:09:20.244787 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.244730 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.245018 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.245001 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 17:09:20.245128 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.245032 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.245387 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.245364 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:09:20.245746 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.245623 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 17:09:20.245746 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.245624 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l4cwr\"" Jul 28 17:09:20.256783 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.256760 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hz9r4" Jul 28 17:09:20.267305 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.267273 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hz9r4" Jul 28 17:09:20.314569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.314497 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 17:09:20.324852 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324784 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-tuned\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.324852 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-cnibin\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.325041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-cni-binary-copy\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.325041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-cni-multus\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.325041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324940 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-slash\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-run-netns\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.324991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-system-cni-dir\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.325041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325027 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-cni-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325064 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-cni-bin\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-node-log\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00143af9-f792-4786-9bd2-aa61b5d1c32a-serviceca\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c41323-027b-4d8e-a1f2-e81e190e4567-tmp\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcc8c290-39e6-4f26-be39-9e67ad8ae399-host-slash\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325277 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd724\" (UniqueName: \"kubernetes.io/projected/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-kube-api-access-jd724\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:20.325347 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-device-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-env-overrides\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c7c6e25-8447-40b7-b2b1-52d9942add01-konnectivity-ca\") pod \"konnectivity-agent-n4s7j\" (UID: \"6c7c6e25-8447-40b7-b2b1-52d9942add01\") " pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26z6\" (UniqueName: \"kubernetes.io/projected/aef8329d-9fa4-4901-b7b4-8dda9ee78061-kube-api-access-w26z6\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-socket-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325566 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-log-socket\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-cni-bin\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-ovnkube-script-lib\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325645 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysctl-d\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.325669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325669 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8tk\" (UniqueName: \"kubernetes.io/projected/91c41323-027b-4d8e-a1f2-e81e190e4567-kube-api-access-ff8tk\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcc8c290-39e6-4f26-be39-9e67ad8ae399-iptables-alerter-script\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00143af9-f792-4786-9bd2-aa61b5d1c32a-serviceca\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-os-release\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325815 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c7c6e25-8447-40b7-b2b1-52d9942add01-agent-certs\") pod \"konnectivity-agent-n4s7j\" (UID: \"6c7c6e25-8447-40b7-b2b1-52d9942add01\") " pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325870 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-modprobe-d\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-kubernetes\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-run\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.325993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-socket-dir-parent\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.325999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c7c6e25-8447-40b7-b2b1-52d9942add01-konnectivity-ca\") pod \"konnectivity-agent-n4s7j\" (UID: \"6c7c6e25-8447-40b7-b2b1-52d9942add01\") " pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-daemon-config\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326034 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef8329d-9fa4-4901-b7b4-8dda9ee78061-tmp-dir\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00143af9-f792-4786-9bd2-aa61b5d1c32a-host\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-host\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-system-cni-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326135 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00143af9-f792-4786-9bd2-aa61b5d1c32a-host\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-os-release\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-k8s-cni-cncf-io\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326232 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-multus-certs\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4997a7f-0501-47fb-ad19-35c54ca8d006-dbus\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-systemd\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cnibin\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326327 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326337 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326355 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mg2\" (UniqueName: \"kubernetes.io/projected/00143af9-f792-4786-9bd2-aa61b5d1c32a-kube-api-access-78mg2\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.326499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysconfig\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-lib-modules\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-registration-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-systemd-units\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326664 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-etc-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326704 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-sys\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-var-lib-kubelet\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-netns\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326783 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4997a7f-0501-47fb-ad19-35c54ca8d006-kubelet-config\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-kubelet\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326859 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cni-binary-copy\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysctl-conf\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326964 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sq22\" (UniqueName: \"kubernetes.io/projected/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-kube-api-access-4sq22\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.327452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.326984 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-conf-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327023 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-sys-fs\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aef8329d-9fa4-4901-b7b4-8dda9ee78061-hosts-file\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-systemd\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327118 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-kubelet\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-hostroot\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327267 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-etc-kubernetes\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-cni-netd\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327321 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-ovnkube-config\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hvn\" (UniqueName: \"kubernetes.io/projected/fcc8c290-39e6-4f26-be39-9e67ad8ae399-kube-api-access-m4hvn\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327371 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf8c\" (UniqueName: \"kubernetes.io/projected/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-kube-api-access-tgf8c\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wql9\" (UniqueName: \"kubernetes.io/projected/f3a990bc-f3a0-4085-b4ad-00317ca7180a-kube-api-access-7wql9\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327425 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-var-lib-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.328284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-ovn\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.329109 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327473 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80259220-717f-491c-8b7d-1c499fac7d57-ovn-node-metrics-cert\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.329109 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-run-ovn-kubernetes\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.329109 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.327523 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9756q\" (UniqueName: \"kubernetes.io/projected/80259220-717f-491c-8b7d-1c499fac7d57-kube-api-access-9756q\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.332553 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.332467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c7c6e25-8447-40b7-b2b1-52d9942add01-agent-certs\") pod \"konnectivity-agent-n4s7j\" (UID: \"6c7c6e25-8447-40b7-b2b1-52d9942add01\") " pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.347425 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.347359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mg2\" (UniqueName: \"kubernetes.io/projected/00143af9-f792-4786-9bd2-aa61b5d1c32a-kube-api-access-78mg2\") pod \"node-ca-z5g7t\" (UID: \"00143af9-f792-4786-9bd2-aa61b5d1c32a\") " pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.421116 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.420986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" event={"ID":"9005259af348bbf3d574131a972e70c3","Type":"ContainerStarted","Data":"8a3fda707e748c126086e47b05d17c70bfce383776d289c85096dd0c50394e2a"} Jul 28 17:09:20.423163 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.423133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" event={"ID":"522862f2528540f980092f7e64425196","Type":"ContainerStarted","Data":"aaa514cb7e5c5005b0c361258dea9b23ee56794ce997f98f1daa9efba8b2ff43"} Jul 28 17:09:20.429181 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysctl-conf\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.429307 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sq22\" (UniqueName: \"kubernetes.io/projected/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-kube-api-access-4sq22\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.429307 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429244 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-conf-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.429307 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.429478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-sys-fs\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.429478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429338 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.429478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429372 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aef8329d-9fa4-4901-b7b4-8dda9ee78061-hosts-file\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.429478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429405 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-systemd\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.429478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-kubelet\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429505 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-hostroot\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-etc-kubernetes\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-cni-netd\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-ovnkube-config\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hvn\" (UniqueName: \"kubernetes.io/projected/fcc8c290-39e6-4f26-be39-9e67ad8ae399-kube-api-access-m4hvn\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf8c\" (UniqueName: \"kubernetes.io/projected/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-kube-api-access-tgf8c\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.429733 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wql9\" (UniqueName: \"kubernetes.io/projected/f3a990bc-f3a0-4085-b4ad-00317ca7180a-kube-api-access-7wql9\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-var-lib-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-ovn\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80259220-717f-491c-8b7d-1c499fac7d57-ovn-node-metrics-cert\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-run-ovn-kubernetes\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9756q\" (UniqueName: \"kubernetes.io/projected/80259220-717f-491c-8b7d-1c499fac7d57-kube-api-access-9756q\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429897 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-tuned\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429947 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-cnibin\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.429982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-cni-binary-copy\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430013 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-cni-multus\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-slash\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430080 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-run-netns\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-system-cni-dir\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.430171 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-cni-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-cni-bin\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-node-log\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c41323-027b-4d8e-a1f2-e81e190e4567-tmp\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430285 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcc8c290-39e6-4f26-be39-9e67ad8ae399-host-slash\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd724\" (UniqueName: \"kubernetes.io/projected/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-kube-api-access-jd724\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-device-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430423 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-env-overrides\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26z6\" (UniqueName: \"kubernetes.io/projected/aef8329d-9fa4-4901-b7b4-8dda9ee78061-kube-api-access-w26z6\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-socket-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-log-socket\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430558 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-cni-bin\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-ovnkube-script-lib\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysctl-d\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8tk\" (UniqueName: \"kubernetes.io/projected/91c41323-027b-4d8e-a1f2-e81e190e4567-kube-api-access-ff8tk\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcc8c290-39e6-4f26-be39-9e67ad8ae399-iptables-alerter-script\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.430824 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430725 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-os-release\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430760 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-modprobe-d\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-kubernetes\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-run\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-socket-dir-parent\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.430999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-daemon-config\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431034 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef8329d-9fa4-4901-b7b4-8dda9ee78061-tmp-dir\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-host\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-system-cni-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-os-release\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431175 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-k8s-cni-cncf-io\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431205 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-multus-certs\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4997a7f-0501-47fb-ad19-35c54ca8d006-dbus\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-systemd\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431315 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cnibin\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431353 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.431842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-node-log\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysctl-conf\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431862 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysconfig\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-lib-modules\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.431995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-registration-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-systemd-units\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-etc-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432090 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-conf-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432108 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432178 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-sys\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-var-lib-kubelet\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-netns\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4997a7f-0501-47fb-ad19-35c54ca8d006-kubelet-config\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-kubelet\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.432775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cni-binary-copy\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432625 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-run-ovn-kubernetes\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432656 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcc8c290-39e6-4f26-be39-9e67ad8ae399-host-slash\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-sys-fs\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432879 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-socket-dir-parent\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.433022 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433030 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef8329d-9fa4-4901-b7b4-8dda9ee78061-tmp-dir\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433047 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cni-binary-copy\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.433105 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.933082197 +0000 UTC m=+2.998578157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-host\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433177 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aef8329d-9fa4-4901-b7b4-8dda9ee78061-hosts-file\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-multus-certs\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-daemon-config\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-cni-multus\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433318 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-systemd\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-slash\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433394 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-run-netns\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.433644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-system-cni-dir\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-systemd\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-multus-cni-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-k8s-cni-cncf-io\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433587 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-cni-bin\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433735 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-cni-binary-copy\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysctl-d\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4997a7f-0501-47fb-ad19-35c54ca8d006-dbus\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-var-lib-kubelet\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.433989 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-var-lib-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.434049 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-ovn\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.434582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.432659 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-cnibin\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.435243 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.434916 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-env-overrides\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.435243 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.434927 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-system-cni-dir\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.435243 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.435180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-cnibin\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436392 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-device-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-hostroot\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-socket-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-etc-kubernetes\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436747 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-log-socket\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436746 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fcc8c290-39e6-4f26-be39-9e67ad8ae399-iptables-alerter-script\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.436784 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-cni-netd\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437076 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c41323-027b-4d8e-a1f2-e81e190e4567-tmp\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-ovnkube-config\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437403 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-host-run-netns\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-tuning-conf-dir\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437492 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-var-lib-kubelet\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437530 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-lib-modules\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437538 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-sysconfig\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437575 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-etc-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-sys\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-os-release\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.438257 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-systemd-units\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437636 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-run-openvswitch\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-cni-bin\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437679 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437687 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-registration-dir\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437714 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437739 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-os-release\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437742 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-kubernetes\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437764 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-modprobe-d\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.437811 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.437866 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.937849484 +0000 UTC m=+3.003345435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.437911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4997a7f-0501-47fb-ad19-35c54ca8d006-kubelet-config\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.438167 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91c41323-027b-4d8e-a1f2-e81e190e4567-run\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.438244 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80259220-717f-491c-8b7d-1c499fac7d57-host-kubelet\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.438303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f3a990bc-f3a0-4085-b4ad-00317ca7180a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.440161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.438879 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80259220-717f-491c-8b7d-1c499fac7d57-ovnkube-script-lib\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.442776 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.442748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80259220-717f-491c-8b7d-1c499fac7d57-ovn-node-metrics-cert\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.443807 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.443665 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:20.443807 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.443690 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:20.443807 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.443704 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:20.443807 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.443771 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:20.943754198 +0000 UTC m=+3.009250165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:20.444590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.444547 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91c41323-027b-4d8e-a1f2-e81e190e4567-etc-tuned\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.447304 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.446802 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sq22\" (UniqueName: \"kubernetes.io/projected/5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924-kube-api-access-4sq22\") pod \"multus-additional-cni-plugins-44qsl\" (UID: \"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924\") " pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.447304 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.447248 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wql9\" (UniqueName: \"kubernetes.io/projected/f3a990bc-f3a0-4085-b4ad-00317ca7180a-kube-api-access-7wql9\") pod \"aws-ebs-csi-driver-node-xjnfs\" (UID: \"f3a990bc-f3a0-4085-b4ad-00317ca7180a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.448430 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.448318 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hvn\" (UniqueName: \"kubernetes.io/projected/fcc8c290-39e6-4f26-be39-9e67ad8ae399-kube-api-access-m4hvn\") pod \"iptables-alerter-jx24d\" (UID: \"fcc8c290-39e6-4f26-be39-9e67ad8ae399\") " pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.450143 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.450097 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9756q\" (UniqueName: \"kubernetes.io/projected/80259220-717f-491c-8b7d-1c499fac7d57-kube-api-access-9756q\") pod \"ovnkube-node-swcbs\" (UID: \"80259220-717f-491c-8b7d-1c499fac7d57\") " pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.450247 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.450225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8tk\" (UniqueName: \"kubernetes.io/projected/91c41323-027b-4d8e-a1f2-e81e190e4567-kube-api-access-ff8tk\") pod \"tuned-2tf6f\" (UID: \"91c41323-027b-4d8e-a1f2-e81e190e4567\") " pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.451232 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.451208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd724\" (UniqueName: \"kubernetes.io/projected/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-kube-api-access-jd724\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:20.451874 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.451823 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26z6\" (UniqueName: \"kubernetes.io/projected/aef8329d-9fa4-4901-b7b4-8dda9ee78061-kube-api-access-w26z6\") pod \"node-resolver-hfc7r\" (UID: \"aef8329d-9fa4-4901-b7b4-8dda9ee78061\") " pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.452280 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.452262 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf8c\" (UniqueName: \"kubernetes.io/projected/bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f-kube-api-access-tgf8c\") pod \"multus-46d7c\" (UID: \"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f\") " pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.525816 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.525556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:20.533177 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.532960 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5g7t" Jul 28 17:09:20.543178 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.543148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46d7c" Jul 28 17:09:20.551004 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.550980 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" Jul 28 17:09:20.558782 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.558717 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jx24d" Jul 28 17:09:20.572995 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.572974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hfc7r" Jul 28 17:09:20.581042 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.581011 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-44qsl" Jul 28 17:09:20.588993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.588970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:20.596724 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.596534 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" Jul 28 17:09:20.936905 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:20.936790 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:20.937061 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.936969 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:20.937104 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:20.937062 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:21.937040424 +0000 UTC m=+4.002536369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:21.037511 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:21.037466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:21.037723 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:21.037543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:21.037723 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.037666 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:21.037723 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.037702 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:21.037723 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.037716 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:21.037976 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.037674 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:21.037976 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.037785 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.037763222 +0000 UTC m=+4.103259194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:21.037976 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.037829 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.037813599 +0000 UTC m=+4.103309545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:21.271285 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:21.271197 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:20 +0000 UTC" deadline="2028-04-17 18:43:40.004414948 +0000 UTC" Jul 28 17:09:21.271285 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:21.271246 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15097h34m18.733188297s" Jul 28 17:09:21.415422 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:21.415390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:21.415594 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.415502 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:21.438156 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.438118 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef8329d_9fa4_4901_b7b4_8dda9ee78061.slice/crio-bd30d77083f791241777612bf413acb1db07edf0cf673b4615bf6988e9b403e9 WatchSource:0}: Error finding container bd30d77083f791241777612bf413acb1db07edf0cf673b4615bf6988e9b403e9: Status 404 returned error can't find the container with id bd30d77083f791241777612bf413acb1db07edf0cf673b4615bf6988e9b403e9 Jul 28 17:09:21.443140 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.443049 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7c6e25_8447_40b7_b2b1_52d9942add01.slice/crio-9847cc2f0c4cfbf09a8334ec60e389d067c5852f74e6b4b3f47bd3750919e8ff WatchSource:0}: Error finding container 9847cc2f0c4cfbf09a8334ec60e389d067c5852f74e6b4b3f47bd3750919e8ff: Status 404 returned error can't find the container with id 9847cc2f0c4cfbf09a8334ec60e389d067c5852f74e6b4b3f47bd3750919e8ff Jul 28 17:09:21.445430 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.445400 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00143af9_f792_4786_9bd2_aa61b5d1c32a.slice/crio-9e3223d0434f76772ab495ba24391f4addbf9958b81d00fa46524c749ca65362 WatchSource:0}: Error finding container 9e3223d0434f76772ab495ba24391f4addbf9958b81d00fa46524c749ca65362: Status 404 returned error can't find the container with id 9e3223d0434f76772ab495ba24391f4addbf9958b81d00fa46524c749ca65362 Jul 28 17:09:21.446188 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.446146 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80259220_717f_491c_8b7d_1c499fac7d57.slice/crio-76afe6c36bf1ed346e8d571cbcc4c50fe0281a43a60ce0237f35995b50641e10 WatchSource:0}: Error finding container 76afe6c36bf1ed346e8d571cbcc4c50fe0281a43a60ce0237f35995b50641e10: Status 404 returned error can't find the container with id 76afe6c36bf1ed346e8d571cbcc4c50fe0281a43a60ce0237f35995b50641e10 Jul 28 17:09:21.447055 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.447027 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a990bc_f3a0_4085_b4ad_00317ca7180a.slice/crio-03b76049335452f08e1ea72f331ffba8a1485476843cb8d0731e9a0a8e75e100 WatchSource:0}: Error finding container 03b76049335452f08e1ea72f331ffba8a1485476843cb8d0731e9a0a8e75e100: Status 404 returned error can't find the container with id 03b76049335452f08e1ea72f331ffba8a1485476843cb8d0731e9a0a8e75e100 Jul 28 17:09:21.447896 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.447873 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc48a1bb_a5b6_4bc5_8c41_8fb2cb77726f.slice/crio-20ad3f274efe11f5532575983acab0fa5ccfa01a11b719f2debc82ba0d59c7a9 WatchSource:0}: Error finding container 20ad3f274efe11f5532575983acab0fa5ccfa01a11b719f2debc82ba0d59c7a9: Status 404 returned error can't find the container with id 20ad3f274efe11f5532575983acab0fa5ccfa01a11b719f2debc82ba0d59c7a9 Jul 28 17:09:21.449753 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.449160 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c41323_027b_4d8e_a1f2_e81e190e4567.slice/crio-d60bb73c73df3d35130ab6b45c27c72768d34b67901f8e328cf6df925bca5f8b WatchSource:0}: Error finding container d60bb73c73df3d35130ab6b45c27c72768d34b67901f8e328cf6df925bca5f8b: Status 404 returned error can't find the container with id d60bb73c73df3d35130ab6b45c27c72768d34b67901f8e328cf6df925bca5f8b Jul 28 17:09:21.451029 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.450412 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc8c290_39e6_4f26_be39_9e67ad8ae399.slice/crio-9f7bca5b507ad17eee375b22257839f14650d5bbdc4f9f15939480df6ac3b0f4 WatchSource:0}: Error finding container 9f7bca5b507ad17eee375b22257839f14650d5bbdc4f9f15939480df6ac3b0f4: Status 404 returned error can't find the container with id 9f7bca5b507ad17eee375b22257839f14650d5bbdc4f9f15939480df6ac3b0f4 Jul 28 17:09:21.451359 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:09:21.451277 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5351fc1f_c8b1_4f82_b5a1_90f8e1f2a924.slice/crio-f6757aa3c2672fb60eca909a692fda1c6e2062698b3e12813112f9841a11965c WatchSource:0}: Error finding container f6757aa3c2672fb60eca909a692fda1c6e2062698b3e12813112f9841a11965c: Status 404 returned error can't find the container with id f6757aa3c2672fb60eca909a692fda1c6e2062698b3e12813112f9841a11965c Jul 28 17:09:21.944481 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:21.944166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:21.944481 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.944453 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:21.944678 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:21.944547 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:23.944523646 +0000 UTC m=+6.010019593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.045062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.045156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.045314 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.045334 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.045348 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.045412 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:24.045392886 +0000 UTC m=+6.110888839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.045991 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:22.046099 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.046048 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:24.046030204 +0000 UTC m=+6.111526153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:22.266928 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.266663 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:09:22.272467 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.272377 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:20 +0000 UTC" deadline="2028-03-27 13:57:30.580244653 +0000 UTC" Jul 28 17:09:22.272467 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.272436 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14588h48m8.307813586s" Jul 28 17:09:22.417588 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.417556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:22.417766 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.417681 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:22.418159 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.418135 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:22.418289 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:22.418245 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:22.451489 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.451452 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" event={"ID":"91c41323-027b-4d8e-a1f2-e81e190e4567","Type":"ContainerStarted","Data":"d60bb73c73df3d35130ab6b45c27c72768d34b67901f8e328cf6df925bca5f8b"} Jul 28 17:09:22.457330 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.457254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" event={"ID":"f3a990bc-f3a0-4085-b4ad-00317ca7180a","Type":"ContainerStarted","Data":"03b76049335452f08e1ea72f331ffba8a1485476843cb8d0731e9a0a8e75e100"} Jul 28 17:09:22.463064 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.463002 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4s7j" event={"ID":"6c7c6e25-8447-40b7-b2b1-52d9942add01","Type":"ContainerStarted","Data":"9847cc2f0c4cfbf09a8334ec60e389d067c5852f74e6b4b3f47bd3750919e8ff"} Jul 28 17:09:22.478000 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.477006 2567 generic.go:356] "Generic (PLEG): container finished" podID="522862f2528540f980092f7e64425196" containerID="d411e87f2bd2a849360a2b0e38bac9611aec97e27cb867caf6ea313676704c6c" exitCode=0 Jul 28 17:09:22.478000 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.477098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" event={"ID":"522862f2528540f980092f7e64425196","Type":"ContainerDied","Data":"d411e87f2bd2a849360a2b0e38bac9611aec97e27cb867caf6ea313676704c6c"} Jul 28 17:09:22.489145 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.489073 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerStarted","Data":"f6757aa3c2672fb60eca909a692fda1c6e2062698b3e12813112f9841a11965c"} Jul 28 17:09:22.499287 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.499254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46d7c" event={"ID":"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f","Type":"ContainerStarted","Data":"20ad3f274efe11f5532575983acab0fa5ccfa01a11b719f2debc82ba0d59c7a9"} Jul 28 17:09:22.512483 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.512455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"76afe6c36bf1ed346e8d571cbcc4c50fe0281a43a60ce0237f35995b50641e10"} Jul 28 17:09:22.519718 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.519687 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5g7t" event={"ID":"00143af9-f792-4786-9bd2-aa61b5d1c32a","Type":"ContainerStarted","Data":"9e3223d0434f76772ab495ba24391f4addbf9958b81d00fa46524c749ca65362"} Jul 28 17:09:22.531623 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.531552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hfc7r" event={"ID":"aef8329d-9fa4-4901-b7b4-8dda9ee78061","Type":"ContainerStarted","Data":"bd30d77083f791241777612bf413acb1db07edf0cf673b4615bf6988e9b403e9"} Jul 28 17:09:22.545232 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.545190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" event={"ID":"9005259af348bbf3d574131a972e70c3","Type":"ContainerStarted","Data":"36cb4be6bbf49092e88ad463cb7ce9afcd61aa48553bb3235fa8cf194328d55e"} Jul 28 17:09:22.554268 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:22.554240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jx24d" event={"ID":"fcc8c290-39e6-4f26-be39-9e67ad8ae399","Type":"ContainerStarted","Data":"9f7bca5b507ad17eee375b22257839f14650d5bbdc4f9f15939480df6ac3b0f4"} Jul 28 17:09:23.415853 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:23.415817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:23.416491 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:23.415990 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:23.565428 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:23.565336 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" event={"ID":"522862f2528540f980092f7e64425196","Type":"ContainerStarted","Data":"de815372c6bc89dc012c34c4c29ab8773179bd49d3160b9f2c37dff23dc3a4c6"} Jul 28 17:09:23.585657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:23.584664 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-110.ec2.internal" podStartSLOduration=3.584641605 podStartE2EDuration="3.584641605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:23.584072246 +0000 UTC m=+5.649568215" watchObservedRunningTime="2026-07-28 17:09:23.584641605 +0000 UTC m=+5.650137574" Jul 28 17:09:23.585657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:23.585100 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-110.ec2.internal" podStartSLOduration=3.585088175 podStartE2EDuration="3.585088175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:22.566321014 +0000 UTC m=+4.631816984" watchObservedRunningTime="2026-07-28 17:09:23.585088175 +0000 UTC m=+5.650584146" Jul 28 17:09:23.962161 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:23.962119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:23.962630 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:23.962606 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:23.962758 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:23.962684 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:27.962664365 +0000 UTC m=+10.028160315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:24.063674 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:24.063635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:24.063903 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:24.063733 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:24.064047 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.064021 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:24.064128 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.064105 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:28.064082406 +0000 UTC m=+10.129578354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:24.064572 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.064545 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:24.064672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.064587 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:24.064672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.064603 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:24.064672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.064666 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:28.064647434 +0000 UTC m=+10.130143394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:24.417912 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:24.417775 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:24.418437 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.417925 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:24.418437 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:24.417988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:24.418437 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:24.418066 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:25.415934 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:25.415872 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:25.416159 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:25.416040 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:26.416019 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:26.415988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:26.416461 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:26.416115 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:26.416536 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:26.416484 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:26.416656 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:26.416575 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:27.415284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:27.415249 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:27.415475 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:27.415400 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:27.997241 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:27.996895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:27.997241 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:27.997066 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:27.997241 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:27.997130 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:35.997110316 +0000 UTC m=+18.062606282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:28.097900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:28.097985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.098008 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.098084 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:36.098064638 +0000 UTC m=+18.163560597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.098100 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.098113 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.098124 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:28.098181 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.098157 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:36.098146924 +0000 UTC m=+18.163642877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:28.416023 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:28.415934 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:28.416173 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.416043 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:28.416537 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:28.416390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:28.416537 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:28.416493 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:29.415402 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:29.415362 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:29.415898 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:29.415505 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:30.415714 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:30.415331 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:30.415714 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:30.415331 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:30.415714 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:30.415439 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:30.415714 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:30.415555 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:31.414971 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:31.414938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:31.415143 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:31.415080 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:32.415072 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:32.414993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:32.415525 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:32.414999 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:32.415525 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:32.415126 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:32.415525 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:32.415203 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:33.415009 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:33.414968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:33.415182 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:33.415097 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:34.415165 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:34.415127 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:34.415634 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:34.415144 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:34.415634 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:34.415264 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:34.415634 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:34.415376 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:35.414980 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:35.414943 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:35.415252 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:35.415071 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:36.059361 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:36.059322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:36.059582 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.059487 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:36.059582 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.059565 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:52.059543698 +0000 UTC m=+34.125039645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:36.160386 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:36.160349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:36.160571 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:36.160416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:36.160571 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.160527 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:36.160571 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.160545 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:36.160571 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.160558 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:36.160571 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.160573 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:36.160823 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.160604 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:52.160584516 +0000 UTC m=+34.226080476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:36.160823 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.160625 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:52.160615659 +0000 UTC m=+34.226111602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:36.415058 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:36.414964 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:36.415223 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:36.414974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:36.415223 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.415105 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:36.415223 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:36.415165 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:37.415474 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:37.415436 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:37.415956 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:37.415577 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:38.416178 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:38.416145 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:38.416629 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:38.416254 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:38.416629 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:38.416306 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:38.416629 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:38.416414 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:39.414801 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:39.414768 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:39.414974 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:39.414902 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:40.415386 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.415248 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:40.415996 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:40.415502 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:40.415996 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.415249 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:40.415996 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:40.415597 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:40.602764 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.602539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46d7c" event={"ID":"bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f","Type":"ContainerStarted","Data":"1bcf46c25e39bcb3f6f3a117130d1d9315369884444e7bc79c3f49f10917e923"} Jul 28 17:09:40.604497 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.604475 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"31dd2002b21043874df79ac15efdb4b8abb4f6558c630212ab1af9cb337f43f0"} Jul 28 17:09:40.604610 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.604549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"d82cf6ca6d80863be086b7e81f8d3d34174fb9115a8a0694e9ea957115eabfba"} Jul 28 17:09:40.604610 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.604604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"e9dac133c31453be7e0357e952e71f6c034495f7c3bf9bdfb63ca967a8a35e7a"} Jul 28 17:09:40.605797 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.605765 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5g7t" event={"ID":"00143af9-f792-4786-9bd2-aa61b5d1c32a","Type":"ContainerStarted","Data":"6be8714c2153126c926d34d64b1562b692e4bc21b5f9602e20e42d2a20664733"} Jul 28 17:09:40.607241 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.607218 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hfc7r" event={"ID":"aef8329d-9fa4-4901-b7b4-8dda9ee78061","Type":"ContainerStarted","Data":"4f605c38b351fd5e5d3ffa140994869ab936a86411989b4d0627aad6b08552ca"} Jul 28 17:09:40.608625 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.608601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" event={"ID":"91c41323-027b-4d8e-a1f2-e81e190e4567","Type":"ContainerStarted","Data":"9fdcc72070375260ce6826760caa7fdf844b6ac2ebb2d851a8a5b7f9faed7ec6"} Jul 28 17:09:40.609842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.609807 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" event={"ID":"f3a990bc-f3a0-4085-b4ad-00317ca7180a","Type":"ContainerStarted","Data":"4035cf7098638bd85baea5dcef052d29999170be6eab295c9643c9e68e310697"} Jul 28 17:09:40.611121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.611094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4s7j" event={"ID":"6c7c6e25-8447-40b7-b2b1-52d9942add01","Type":"ContainerStarted","Data":"e47758f79c2d9d78fce7879377e3d99337c3c58a8927b153e10154d336e9a02b"} Jul 28 17:09:40.612464 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.612443 2567 generic.go:356] "Generic (PLEG): container finished" podID="5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924" containerID="1a8cbb0f51ce507291ef98c142d3d34ec8b4224fa5207989adb0b8e167a0c2ce" exitCode=0 Jul 28 17:09:40.612556 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.612470 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerDied","Data":"1a8cbb0f51ce507291ef98c142d3d34ec8b4224fa5207989adb0b8e167a0c2ce"} Jul 28 17:09:40.620565 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.620521 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-46d7c" podStartSLOduration=3.8537656240000002 podStartE2EDuration="22.620506334s" totalImagesPullingTime="18.76674071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.620227128 +0000 UTC m=+22.685723093" watchObservedRunningTime="2026-07-28 17:09:40.620506334 +0000 UTC m=+22.686002300" Jul 28 17:09:40.637200 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.637163 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hfc7r" podStartSLOduration=2.922695874 podStartE2EDuration="21.637150724s" totalImagesPullingTime="18.71445485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.637058364 +0000 UTC m=+22.702554329" watchObservedRunningTime="2026-07-28 17:09:40.637150724 +0000 UTC m=+22.702646667" Jul 28 17:09:40.689509 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.689462 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n4s7j" podStartSLOduration=8.198082931 podStartE2EDuration="22.689449608s" totalImagesPullingTime="14.491366677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.688993131 +0000 UTC m=+22.754489096" watchObservedRunningTime="2026-07-28 17:09:40.689449608 +0000 UTC m=+22.754945573" Jul 28 17:09:40.707907 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:40.707850 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z5g7t" podStartSLOduration=4.000725581 podStartE2EDuration="22.707835427s" totalImagesPullingTime="18.707109846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.707488138 +0000 UTC m=+22.772984104" watchObservedRunningTime="2026-07-28 17:09:40.707835427 +0000 UTC m=+22.773331391" Jul 28 17:09:41.415374 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.415337 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:41.415622 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:41.415487 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:41.614538 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.614512 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 17:09:41.618822 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.618782 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"d9b33be096d6317b1a2ed39eda28b92969079cd1bdeef66f2b7dcac8e6b929f9"} Jul 28 17:09:41.618947 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.618894 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"c6a7538b7310c0b71db883e0b556c1f4b9da0d64fb36bc3cf52f3a6f9e75e398"} Jul 28 17:09:41.618997 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.618944 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"c0d292ca460cef331b97facf284cb0572a42f83bc2b0064c6f456b0fd9da3b8b"} Jul 28 17:09:41.620369 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.620334 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jx24d" event={"ID":"fcc8c290-39e6-4f26-be39-9e67ad8ae399","Type":"ContainerStarted","Data":"efb3a944ca268d310d77a8ba0795b77e2ee178c6cc53915b8fbcd8413669c0cd"} Jul 28 17:09:41.622252 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.622228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" event={"ID":"f3a990bc-f3a0-4085-b4ad-00317ca7180a","Type":"ContainerStarted","Data":"00331ebce7abdba902bcc6d461320ac9e401d7aa161110720c3230ef4466cdae"} Jul 28 17:09:41.636590 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.636548 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tf6f" podStartSLOduration=3.932438501 podStartE2EDuration="22.636533769s" totalImagesPullingTime="18.704095268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.730687748 +0000 UTC m=+22.796183713" watchObservedRunningTime="2026-07-28 17:09:41.636533769 +0000 UTC m=+23.702029735" Jul 28 17:09:41.636876 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:41.636834 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jx24d" podStartSLOduration=3.95903135 podStartE2EDuration="22.636825932s" totalImagesPullingTime="18.677794582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:41.636158925 +0000 UTC m=+23.701654891" watchObservedRunningTime="2026-07-28 17:09:41.636825932 +0000 UTC m=+23.702321897" Jul 28 17:09:42.291223 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:42.291123 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:09:41.614534366Z","UUID":"074b3b4c-f330-4c77-aacd-ae42c633707e","Handler":null,"Name":"","Endpoint":""} Jul 28 17:09:42.293112 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:42.293089 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 17:09:42.293254 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:42.293144 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 17:09:42.415388 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:42.415361 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:42.415571 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:42.415495 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:42.415640 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:42.415602 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:42.416031 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:42.415709 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:43.003327 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.003292 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:43.004080 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.004057 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:43.415219 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.415192 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:43.415409 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:43.415304 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:43.630385 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.630342 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"ef4c3e1c933f8ae34f396645488c602ef3655095012fa2e71acb19f23c0edd93"} Jul 28 17:09:43.632454 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.632406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" event={"ID":"f3a990bc-f3a0-4085-b4ad-00317ca7180a","Type":"ContainerStarted","Data":"c73b8a42689ae7201ca23c0835fe75b15b50124945e928b740591270822d173e"} Jul 28 17:09:43.632817 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.632797 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:43.633306 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.633287 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n4s7j" Jul 28 17:09:43.653263 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:43.653210 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjnfs" podStartSLOduration=3.625502369 podStartE2EDuration="24.653193023s" totalImagesPullingTime="21.027690654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:43.651677426 +0000 UTC m=+25.717173391" watchObservedRunningTime="2026-07-28 17:09:43.653193023 +0000 UTC m=+25.718688988" Jul 28 17:09:44.415179 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:44.414977 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:44.415350 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:44.414980 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:44.415350 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:44.415260 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:44.415458 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:44.415414 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:45.415685 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.415520 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:45.416279 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:45.415762 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:45.643154 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.643118 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" event={"ID":"80259220-717f-491c-8b7d-1c499fac7d57","Type":"ContainerStarted","Data":"03b05b350281b630d0303a360516741cde0fc54126efa999862686b9a5421e6a"} Jul 28 17:09:45.643353 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.643329 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:45.643420 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.643408 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:45.644991 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.644967 2567 generic.go:356] "Generic (PLEG): container finished" podID="5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924" containerID="bb5f436985650abd1a26a4f1af412d48a6622463a6da1819c8795a47a692b12a" exitCode=0 Jul 28 17:09:45.645099 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.645050 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerDied","Data":"bb5f436985650abd1a26a4f1af412d48a6622463a6da1819c8795a47a692b12a"} Jul 28 17:09:45.659844 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.659822 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:45.671081 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:45.670998 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" podStartSLOduration=7.90348203 podStartE2EDuration="26.670986678s" totalImagesPullingTime="18.767504648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:45.669515606 +0000 UTC m=+27.735011572" watchObservedRunningTime="2026-07-28 17:09:45.670986678 +0000 UTC m=+27.736482640" Jul 28 17:09:46.415380 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:46.415212 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:46.415380 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:46.415229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:46.415380 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:46.415321 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:46.415724 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:46.415421 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:46.647799 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:46.647773 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:46.662676 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:46.662653 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:09:47.044764 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.044493 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2s7mz"] Jul 28 17:09:47.044915 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.044884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:47.045021 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:47.044992 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:47.045357 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.045327 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2bfzx"] Jul 28 17:09:47.045463 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.045427 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:47.045544 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:47.045510 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:47.045951 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.045931 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nlxkb"] Jul 28 17:09:47.046058 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.046045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:47.046163 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:47.046146 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:47.651351 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.651311 2567 generic.go:356] "Generic (PLEG): container finished" podID="5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924" containerID="caa92388417a5af373dd3613ec6677da785a4224483625a0118f8777e475d27e" exitCode=0 Jul 28 17:09:47.651801 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:47.651391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerDied","Data":"caa92388417a5af373dd3613ec6677da785a4224483625a0118f8777e475d27e"} Jul 28 17:09:48.416874 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:48.416820 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:48.417089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:48.416934 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:48.417089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:48.416970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:48.417089 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:48.416981 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:48.417089 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:48.417033 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:48.417313 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:48.417138 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:49.658616 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:49.658586 2567 generic.go:356] "Generic (PLEG): container finished" podID="5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924" containerID="6df04faccecfc41154b5dbc6c4f6bc3744e1c238ad8425184aa5177df73e9c7a" exitCode=0 Jul 28 17:09:49.659152 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:49.658647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerDied","Data":"6df04faccecfc41154b5dbc6c4f6bc3744e1c238ad8425184aa5177df73e9c7a"} Jul 28 17:09:50.415809 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:50.415772 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:50.416005 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:50.415779 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:50.416005 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:50.415921 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:50.416005 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:50.415979 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:50.416130 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:50.416096 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:50.416233 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:50.416203 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:52.082538 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:52.082447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:52.082945 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.082631 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:52.082945 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.082714 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret podName:e4997a7f-0501-47fb-ad19-35c54ca8d006 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:24.082691432 +0000 UTC m=+66.148187388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret") pod "global-pull-secret-syncer-2s7mz" (UID: "e4997a7f-0501-47fb-ad19-35c54ca8d006") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:09:52.183357 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:52.183319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:52.183528 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:52.183391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:52.183528 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.183502 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:52.183528 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.183521 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:09:52.183660 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.183540 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:09:52.183660 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.183551 2567 projected.go:196] Error preparing data for projected volume kube-api-access-t9mzw for pod openshift-network-diagnostics/network-check-target-2bfzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:52.183660 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.183580 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs podName:acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:24.183557287 +0000 UTC m=+66.249053236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs") pod "network-metrics-daemon-nlxkb" (UID: "acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:09:52.183660 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.183599 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw podName:0923ada6-bef2-4608-a197-021abbcacd6d nodeName:}" failed. No retries permitted until 2026-07-28 17:10:24.183591092 +0000 UTC m=+66.249087036 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9mzw" (UniqueName: "kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw") pod "network-check-target-2bfzx" (UID: "0923ada6-bef2-4608-a197-021abbcacd6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:09:52.415034 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:52.414995 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:52.415229 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:52.415007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:52.415229 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.415127 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:52.415229 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:52.415141 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:52.415394 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.415233 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:52.415394 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:52.415324 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:53.671342 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:53.671304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerStarted","Data":"82eecc63191b9b63f6ebcb394697c6bec7dffc57c7c5387cfe9ed962b5d29dc7"} Jul 28 17:09:54.415215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:54.415176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:54.415416 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:54.415302 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:54.415416 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:54.415313 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:54.415416 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:54.415398 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:54.415592 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:54.415449 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:54.415592 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:54.415502 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:54.676350 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:54.676265 2567 generic.go:356] "Generic (PLEG): container finished" podID="5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924" containerID="82eecc63191b9b63f6ebcb394697c6bec7dffc57c7c5387cfe9ed962b5d29dc7" exitCode=0 Jul 28 17:09:54.676350 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:54.676328 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerDied","Data":"82eecc63191b9b63f6ebcb394697c6bec7dffc57c7c5387cfe9ed962b5d29dc7"} Jul 28 17:09:55.681597 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:55.681342 2567 generic.go:356] "Generic (PLEG): container finished" podID="5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924" containerID="b0f113b06125dfbd7e2706f74df4b4a9057c24a787da9f56eb238db316fe0f47" exitCode=0 Jul 28 17:09:55.681998 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:55.681415 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerDied","Data":"b0f113b06125dfbd7e2706f74df4b4a9057c24a787da9f56eb238db316fe0f47"} Jul 28 17:09:56.415142 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:56.415111 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:56.415299 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:56.415111 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:56.415299 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:56.415220 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:56.415299 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:56.415224 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:56.415299 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:56.415283 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:56.415476 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:56.415396 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:09:56.687120 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:56.687046 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-44qsl" event={"ID":"5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924","Type":"ContainerStarted","Data":"4df508fce20afdd0af6456e56813813f2c89dcdfc98bb26897315ac669114f81"} Jul 28 17:09:56.719512 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:56.719456 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-44qsl" podStartSLOduration=8.597434569 podStartE2EDuration="37.719439755s" totalImagesPullingTime="29.122005186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:56.717539586 +0000 UTC m=+38.783035551" watchObservedRunningTime="2026-07-28 17:09:56.719439755 +0000 UTC m=+38.784935721" Jul 28 17:09:58.416429 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:58.416395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:09:58.416906 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:58.416480 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:09:58.416906 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:58.416575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:09:58.416906 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:09:58.416678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:09:58.416906 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:58.416685 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:09:58.416906 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:09:58.416764 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:10:00.415612 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:00.415577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:00.415993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:00.415577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:00.415993 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:00.415682 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:10:00.415993 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:00.415709 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:00.415993 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:00.415792 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:10:00.415993 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:00.415848 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:10:02.414906 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:02.414874 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:02.415388 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:02.414855 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:02.415388 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:02.414984 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:02.415388 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:02.414985 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:10:02.415388 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:02.415053 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:10:02.415388 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:02.415128 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:10:04.415398 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:04.415359 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:04.416015 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:04.415365 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:04.416015 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:04.415453 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:10:04.416015 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:04.415518 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:10:04.416015 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:04.415565 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:04.416015 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:04.415619 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:10:06.415614 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:06.415562 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:06.415614 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:06.415591 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:06.416156 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:06.415562 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:06.416156 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:06.415685 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:10:06.416156 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:06.415745 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:10:06.416156 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:06.415811 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:10:08.416484 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:08.416448 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:08.416879 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:08.416553 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:08.416879 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:08.416595 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:08.416879 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:08.416589 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlxkb" podUID="acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5" Jul 28 17:10:08.416879 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:08.416669 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2bfzx" podUID="0923ada6-bef2-4608-a197-021abbcacd6d" Jul 28 17:10:08.416879 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:08.416763 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2s7mz" podUID="e4997a7f-0501-47fb-ad19-35c54ca8d006" Jul 28 17:10:09.764244 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.764215 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-110.ec2.internal" event="NodeReady" Jul 28 17:10:09.764636 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.764344 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 17:10:09.820930 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.820849 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f6894798-z2wwk"] Jul 28 17:10:09.836043 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.836014 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dp5lc"] Jul 28 17:10:09.836181 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.836121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.839812 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.839793 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hm5wg\"" Jul 28 17:10:09.840458 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.840430 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 17:10:09.840585 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.840502 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 17:10:09.840681 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.840667 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 17:10:09.846677 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.846632 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 17:10:09.850119 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.850099 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-622rh"] Jul 28 17:10:09.850256 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.850235 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:09.853088 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.853072 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 17:10:09.853267 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.853255 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtzb7\"" Jul 28 17:10:09.857970 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.857952 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 17:10:09.872486 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.872464 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:09.872715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.872694 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f6894798-z2wwk"] Jul 28 17:10:09.872780 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.872749 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dp5lc"] Jul 28 17:10:09.872829 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.872790 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-622rh"] Jul 28 17:10:09.876392 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.876371 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 17:10:09.876478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.876440 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h6cgw\"" Jul 28 17:10:09.876478 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.876373 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 17:10:09.876569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.876486 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 17:10:09.911215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52c988b9-2f97-47df-adf1-dda8d949890a-registry-certificates\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-config-volume\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:09.911334 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911234 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52c988b9-2f97-47df-adf1-dda8d949890a-image-registry-private-configuration\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911334 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911256 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52c988b9-2f97-47df-adf1-dda8d949890a-ca-trust-extracted\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911334 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52c988b9-2f97-47df-adf1-dda8d949890a-trusted-ca\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911334 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-metrics-tls\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:09.911451 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911336 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-tmp-dir\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:09.911451 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4bdn\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-kube-api-access-z4bdn\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911451 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rqn\" (UniqueName: \"kubernetes.io/projected/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-kube-api-access-f6rqn\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:09.911451 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911439 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzptn\" (UniqueName: \"kubernetes.io/projected/5bb2f701-67fa-48f1-9afd-0c7ecd04039a-kube-api-access-fzptn\") pod \"ingress-canary-622rh\" (UID: \"5bb2f701-67fa-48f1-9afd-0c7ecd04039a\") " pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:09.911571 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-registry-tls\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911571 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52c988b9-2f97-47df-adf1-dda8d949890a-installation-pull-secrets\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911571 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-bound-sa-token\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:09.911571 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:09.911550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bb2f701-67fa-48f1-9afd-0c7ecd04039a-cert\") pod \"ingress-canary-622rh\" (UID: \"5bb2f701-67fa-48f1-9afd-0c7ecd04039a\") " pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:10.012150 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-config-volume\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.012343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52c988b9-2f97-47df-adf1-dda8d949890a-image-registry-private-configuration\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52c988b9-2f97-47df-adf1-dda8d949890a-ca-trust-extracted\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52c988b9-2f97-47df-adf1-dda8d949890a-trusted-ca\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012508 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-metrics-tls\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.012508 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012369 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-tmp-dir\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.012508 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4bdn\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-kube-api-access-z4bdn\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012651 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rqn\" (UniqueName: \"kubernetes.io/projected/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-kube-api-access-f6rqn\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.012651 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzptn\" (UniqueName: \"kubernetes.io/projected/5bb2f701-67fa-48f1-9afd-0c7ecd04039a-kube-api-access-fzptn\") pod \"ingress-canary-622rh\" (UID: \"5bb2f701-67fa-48f1-9afd-0c7ecd04039a\") " pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:10.012651 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012643 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-registry-tls\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012794 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012672 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52c988b9-2f97-47df-adf1-dda8d949890a-installation-pull-secrets\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012794 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012739 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-bound-sa-token\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012794 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012741 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52c988b9-2f97-47df-adf1-dda8d949890a-ca-trust-extracted\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.012794 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bb2f701-67fa-48f1-9afd-0c7ecd04039a-cert\") pod \"ingress-canary-622rh\" (UID: \"5bb2f701-67fa-48f1-9afd-0c7ecd04039a\") " pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:10.013034 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52c988b9-2f97-47df-adf1-dda8d949890a-registry-certificates\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.013034 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012835 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-config-volume\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.013034 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.012746 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-tmp-dir\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.013613 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.013593 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52c988b9-2f97-47df-adf1-dda8d949890a-registry-certificates\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.016645 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.016621 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-metrics-tls\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.016739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.016647 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52c988b9-2f97-47df-adf1-dda8d949890a-installation-pull-secrets\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.016739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.016683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-registry-tls\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.016739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.016706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bb2f701-67fa-48f1-9afd-0c7ecd04039a-cert\") pod \"ingress-canary-622rh\" (UID: \"5bb2f701-67fa-48f1-9afd-0c7ecd04039a\") " pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:10.016739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.016710 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52c988b9-2f97-47df-adf1-dda8d949890a-image-registry-private-configuration\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.020893 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.020852 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52c988b9-2f97-47df-adf1-dda8d949890a-trusted-ca\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.021244 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.021226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rqn\" (UniqueName: \"kubernetes.io/projected/fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a-kube-api-access-f6rqn\") pod \"dns-default-dp5lc\" (UID: \"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a\") " pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.021426 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.021408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4bdn\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-kube-api-access-z4bdn\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.021577 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.021559 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzptn\" (UniqueName: \"kubernetes.io/projected/5bb2f701-67fa-48f1-9afd-0c7ecd04039a-kube-api-access-fzptn\") pod \"ingress-canary-622rh\" (UID: \"5bb2f701-67fa-48f1-9afd-0c7ecd04039a\") " pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:10.022051 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.022031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52c988b9-2f97-47df-adf1-dda8d949890a-bound-sa-token\") pod \"image-registry-78f6894798-z2wwk\" (UID: \"52c988b9-2f97-47df-adf1-dda8d949890a\") " pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.146315 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.146271 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.158433 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.158407 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:10.181734 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.181709 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-622rh" Jul 28 17:10:10.336017 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.335878 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f454b754-6629g"] Jul 28 17:10:10.356153 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.356124 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f454b754-6629g"] Jul 28 17:10:10.356311 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.356240 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.359524 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.359496 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-622rh"] Jul 28 17:10:10.360660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.360639 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 17:10:10.360769 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.360643 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 17:10:10.360769 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.360733 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 17:10:10.360925 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.360770 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v7zdk\"" Jul 28 17:10:10.360925 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.360833 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 17:10:10.361784 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.361754 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 17:10:10.361784 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.361772 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 17:10:10.362140 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.362124 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 17:10:10.363393 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:10.363370 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb2f701_67fa_48f1_9afd_0c7ecd04039a.slice/crio-b59c86e6ec71e29b44fbc76398967bb5d0ffe67c12b4d735f405bf7a77437bac WatchSource:0}: Error finding container b59c86e6ec71e29b44fbc76398967bb5d0ffe67c12b4d735f405bf7a77437bac: Status 404 returned error can't find the container with id b59c86e6ec71e29b44fbc76398967bb5d0ffe67c12b4d735f405bf7a77437bac Jul 28 17:10:10.372319 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.372299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f6894798-z2wwk"] Jul 28 17:10:10.375252 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:10.375228 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c988b9_2f97_47df_adf1_dda8d949890a.slice/crio-156ff2a9d2f9fa751b6510b8b3c862f1eccc15cad9bf26bba005a2ca526a9cf8 WatchSource:0}: Error finding container 156ff2a9d2f9fa751b6510b8b3c862f1eccc15cad9bf26bba005a2ca526a9cf8: Status 404 returned error can't find the container with id 156ff2a9d2f9fa751b6510b8b3c862f1eccc15cad9bf26bba005a2ca526a9cf8 Jul 28 17:10:10.376694 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.376674 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dp5lc"] Jul 28 17:10:10.379781 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:10.379761 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2cdbca_4cc3_4e78_bacc_9481f8fc9c5a.slice/crio-622be03b1d2a60d778474127b3f927205fb78e19a2fe4a7d4360e250ad5cb122 WatchSource:0}: Error finding container 622be03b1d2a60d778474127b3f927205fb78e19a2fe4a7d4360e250ad5cb122: Status 404 returned error can't find the container with id 622be03b1d2a60d778474127b3f927205fb78e19a2fe4a7d4360e250ad5cb122 Jul 28 17:10:10.415854 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.415807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:10.415854 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.415826 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vj4\" (UniqueName: \"kubernetes.io/projected/03030ad1-2da2-49f0-85c0-93ea9ead8259-kube-api-access-f4vj4\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.415996 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.415807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:10.415996 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.415924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-oauth-serving-cert\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.415996 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.415807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:10.415996 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.415966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-config\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.416176 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.416044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-oauth-config\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.416176 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.416078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-serving-cert\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.416176 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.416104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-service-ca\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.419059 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.419023 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:10:10.419196 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.419156 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4dt6\"" Jul 28 17:10:10.419416 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.419310 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:10:10.419416 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.419333 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:10:10.419416 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.419392 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:10:10.419675 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.419652 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7zr5q\"" Jul 28 17:10:10.516768 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.516740 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-oauth-serving-cert\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.516901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.516773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-config\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.516901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.516810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-oauth-config\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.516901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.516833 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-serving-cert\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.516901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.516856 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-service-ca\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.517104 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.516917 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vj4\" (UniqueName: \"kubernetes.io/projected/03030ad1-2da2-49f0-85c0-93ea9ead8259-kube-api-access-f4vj4\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.517546 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.517523 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-config\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.517647 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.517545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-oauth-serving-cert\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.517647 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.517549 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-service-ca\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.519120 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.519092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-serving-cert\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.519366 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.519351 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-oauth-config\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.525773 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.525756 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vj4\" (UniqueName: \"kubernetes.io/projected/03030ad1-2da2-49f0-85c0-93ea9ead8259-kube-api-access-f4vj4\") pod \"console-86f454b754-6629g\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.667104 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.667007 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:10.720653 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.720604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-622rh" event={"ID":"5bb2f701-67fa-48f1-9afd-0c7ecd04039a","Type":"ContainerStarted","Data":"b59c86e6ec71e29b44fbc76398967bb5d0ffe67c12b4d735f405bf7a77437bac"} Jul 28 17:10:10.724144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.724065 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" event={"ID":"52c988b9-2f97-47df-adf1-dda8d949890a","Type":"ContainerStarted","Data":"e09aec5de70c71654d62d8d8dce2d7d4d1005ab9af162e0a3698f72249b8e639"} Jul 28 17:10:10.724144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.724115 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" event={"ID":"52c988b9-2f97-47df-adf1-dda8d949890a","Type":"ContainerStarted","Data":"156ff2a9d2f9fa751b6510b8b3c862f1eccc15cad9bf26bba005a2ca526a9cf8"} Jul 28 17:10:10.725053 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.724575 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:10.725321 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.725296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dp5lc" event={"ID":"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a","Type":"ContainerStarted","Data":"622be03b1d2a60d778474127b3f927205fb78e19a2fe4a7d4360e250ad5cb122"} Jul 28 17:10:10.829115 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.829055 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" podStartSLOduration=1.829034199 podStartE2EDuration="1.829034199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:10.754717231 +0000 UTC m=+52.820213197" watchObservedRunningTime="2026-07-28 17:10:10.829034199 +0000 UTC m=+52.894530168" Jul 28 17:10:10.830089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:10.830066 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f454b754-6629g"] Jul 28 17:10:10.833326 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:10.833299 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03030ad1_2da2_49f0_85c0_93ea9ead8259.slice/crio-161d2afeb32196f53378cce61f8d2451984d79b64bf31eb56effe7cfce4c344a WatchSource:0}: Error finding container 161d2afeb32196f53378cce61f8d2451984d79b64bf31eb56effe7cfce4c344a: Status 404 returned error can't find the container with id 161d2afeb32196f53378cce61f8d2451984d79b64bf31eb56effe7cfce4c344a Jul 28 17:10:11.730005 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:11.729964 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f454b754-6629g" event={"ID":"03030ad1-2da2-49f0-85c0-93ea9ead8259","Type":"ContainerStarted","Data":"161d2afeb32196f53378cce61f8d2451984d79b64bf31eb56effe7cfce4c344a"} Jul 28 17:10:12.500706 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.500656 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d99df865-jkjfs"] Jul 28 17:10:12.514317 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.514292 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.516753 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.516602 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d99df865-jkjfs"] Jul 28 17:10:12.524186 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.524163 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 17:10:12.532033 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-service-ca\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.532157 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-config\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.532157 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-oauth-serving-cert\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.532274 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wbs\" (UniqueName: \"kubernetes.io/projected/6ddc3d4f-180d-47b9-868f-356568a5cf38-kube-api-access-d5wbs\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.532274 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532192 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-trusted-ca-bundle\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.532274 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-oauth-config\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.532419 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.532300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-serving-cert\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633676 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-service-ca\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633676 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-config\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-oauth-serving-cert\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wbs\" (UniqueName: \"kubernetes.io/projected/6ddc3d4f-180d-47b9-868f-356568a5cf38-kube-api-access-d5wbs\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-trusted-ca-bundle\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-oauth-config\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.633962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.633834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-serving-cert\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.634506 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.634480 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-service-ca\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.634615 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.634585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-oauth-serving-cert\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.634977 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.634951 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-trusted-ca-bundle\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.635084 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.635067 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-config\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.636931 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.636904 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-oauth-config\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.637314 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.637291 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-serving-cert\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.643805 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.643460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wbs\" (UniqueName: \"kubernetes.io/projected/6ddc3d4f-180d-47b9-868f-356568a5cf38-kube-api-access-d5wbs\") pod \"console-54d99df865-jkjfs\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:12.826434 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:12.826349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:13.482820 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.482782 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d99df865-jkjfs"] Jul 28 17:10:13.488907 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:13.488843 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddc3d4f_180d_47b9_868f_356568a5cf38.slice/crio-5e964945a187df6156f7bdeeb4a3cbc9d55df99a8daf11666787f86b8fe50fe9 WatchSource:0}: Error finding container 5e964945a187df6156f7bdeeb4a3cbc9d55df99a8daf11666787f86b8fe50fe9: Status 404 returned error can't find the container with id 5e964945a187df6156f7bdeeb4a3cbc9d55df99a8daf11666787f86b8fe50fe9 Jul 28 17:10:13.738183 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.738149 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dp5lc" event={"ID":"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a","Type":"ContainerStarted","Data":"5710faf6cccb30b930ac84f4c3302007fa8c694de070bae7f173c090198d2825"} Jul 28 17:10:13.738657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.738221 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dp5lc" event={"ID":"fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a","Type":"ContainerStarted","Data":"d0f2234030f3cae432674697118cc5aa0a0e8afc912425848ddc3d0b84588686"} Jul 28 17:10:13.738657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.738248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:13.739728 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.739642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d99df865-jkjfs" event={"ID":"6ddc3d4f-180d-47b9-868f-356568a5cf38","Type":"ContainerStarted","Data":"5e964945a187df6156f7bdeeb4a3cbc9d55df99a8daf11666787f86b8fe50fe9"} Jul 28 17:10:13.741439 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.741262 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-622rh" event={"ID":"5bb2f701-67fa-48f1-9afd-0c7ecd04039a","Type":"ContainerStarted","Data":"005e663af0ac2315abac33ad08042664822ed51828391f14fbac3700dba9bdc9"} Jul 28 17:10:13.760133 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.760069 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dp5lc" podStartSLOduration=1.81356831 podStartE2EDuration="4.760050655s" totalImagesPullingTime="2.946482345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:13.758459562 +0000 UTC m=+55.823955541" watchObservedRunningTime="2026-07-28 17:10:13.760050655 +0000 UTC m=+55.825546618" Jul 28 17:10:13.777536 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:13.777484 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-622rh" podStartSLOduration=1.80526611 podStartE2EDuration="4.777469631s" totalImagesPullingTime="2.972203521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:13.776763548 +0000 UTC m=+55.842259514" watchObservedRunningTime="2026-07-28 17:10:13.777469631 +0000 UTC m=+55.842965596" Jul 28 17:10:16.753676 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:16.753639 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f454b754-6629g" event={"ID":"03030ad1-2da2-49f0-85c0-93ea9ead8259","Type":"ContainerStarted","Data":"0b4c8b4227ddd392d22fe251661cd5404cc076eb3e85dfd90a303921d7a30b2f"} Jul 28 17:10:16.755168 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:16.755138 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d99df865-jkjfs" event={"ID":"6ddc3d4f-180d-47b9-868f-356568a5cf38","Type":"ContainerStarted","Data":"b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d"} Jul 28 17:10:16.776529 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:16.776490 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86f454b754-6629g" podStartSLOduration=1.524487157 podStartE2EDuration="6.776476979s" totalImagesPullingTime="5.251989822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:16.775712197 +0000 UTC m=+58.841208161" watchObservedRunningTime="2026-07-28 17:10:16.776476979 +0000 UTC m=+58.841972944" Jul 28 17:10:16.799992 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:16.799955 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54d99df865-jkjfs" podStartSLOduration=2.188212325 podStartE2EDuration="4.799939994s" totalImagesPullingTime="2.611727669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:16.798852363 +0000 UTC m=+58.864348326" watchObservedRunningTime="2026-07-28 17:10:16.799939994 +0000 UTC m=+58.865435961" Jul 28 17:10:17.743618 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.743583 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj"] Jul 28 17:10:17.745659 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.745642 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:17.748774 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.748754 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 17:10:17.748947 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.748931 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wsf68\"" Jul 28 17:10:17.759903 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.759386 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj"] Jul 28 17:10:17.772080 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.772055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/366e1152-89b9-42a5-b134-e23a4ffb464f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5xqpj\" (UID: \"366e1152-89b9-42a5-b134-e23a4ffb464f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:17.872740 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.872699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/366e1152-89b9-42a5-b134-e23a4ffb464f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5xqpj\" (UID: \"366e1152-89b9-42a5-b134-e23a4ffb464f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:17.875176 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:17.875149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/366e1152-89b9-42a5-b134-e23a4ffb464f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5xqpj\" (UID: \"366e1152-89b9-42a5-b134-e23a4ffb464f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:18.054017 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.053926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:18.176731 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.176693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj"] Jul 28 17:10:18.179505 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:18.179474 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366e1152_89b9_42a5_b134_e23a4ffb464f.slice/crio-ae0aafd66223070f568c40963e4c4b7f2c735a9ebb198e430d749171369c1350 WatchSource:0}: Error finding container ae0aafd66223070f568c40963e4c4b7f2c735a9ebb198e430d749171369c1350: Status 404 returned error can't find the container with id ae0aafd66223070f568c40963e4c4b7f2c735a9ebb198e430d749171369c1350 Jul 28 17:10:18.518278 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.518237 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zlnz9"] Jul 28 17:10:18.521000 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.520972 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.525182 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.525165 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 17:10:18.526193 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.526177 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 17:10:18.526273 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.526223 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 17:10:18.526330 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.526231 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs9nh\"" Jul 28 17:10:18.529041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.529024 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 17:10:18.542172 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.542153 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zlnz9"] Jul 28 17:10:18.578846 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.578816 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbf8q\" (UniqueName: \"kubernetes.io/projected/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-api-access-vbf8q\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.578976 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.578855 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf7c933c-1dcc-4849-80b8-c5a349f19008-data-volume\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.578976 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.578941 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf7c933c-1dcc-4849-80b8-c5a349f19008-crio-socket\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.579043 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.578987 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.579043 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.579011 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.665627 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.665601 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-swcbs" Jul 28 17:10:18.679952 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.679927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf7c933c-1dcc-4849-80b8-c5a349f19008-crio-socket\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.680056 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.679975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.680056 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.680006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.680056 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.680040 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf7c933c-1dcc-4849-80b8-c5a349f19008-crio-socket\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.680210 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:18.680119 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:10:18.680210 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:18.680147 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:10:18.680210 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:18.680191 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm podName:bf7c933c-1dcc-4849-80b8-c5a349f19008 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.180172776 +0000 UTC m=+61.245668728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlnz9" (UID: "bf7c933c-1dcc-4849-80b8-c5a349f19008") : configmap "kube-rbac-proxy" not found Jul 28 17:10:18.680365 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:18.680215 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls podName:bf7c933c-1dcc-4849-80b8-c5a349f19008 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.180197825 +0000 UTC m=+61.245693791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlnz9" (UID: "bf7c933c-1dcc-4849-80b8-c5a349f19008") : secret "insights-runtime-extractor-tls" not found Jul 28 17:10:18.680365 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.680239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbf8q\" (UniqueName: \"kubernetes.io/projected/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-api-access-vbf8q\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.680365 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.680294 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf7c933c-1dcc-4849-80b8-c5a349f19008-data-volume\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.680562 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.680545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf7c933c-1dcc-4849-80b8-c5a349f19008-data-volume\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.691197 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.691174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbf8q\" (UniqueName: \"kubernetes.io/projected/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-api-access-vbf8q\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:18.766365 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:18.766330 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" event={"ID":"366e1152-89b9-42a5-b134-e23a4ffb464f","Type":"ContainerStarted","Data":"ae0aafd66223070f568c40963e4c4b7f2c735a9ebb198e430d749171369c1350"} Jul 28 17:10:19.183308 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:19.183264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:19.183308 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:19.183307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:19.183519 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:19.183423 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:10:19.183519 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:19.183424 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:10:19.183519 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:19.183485 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm podName:bf7c933c-1dcc-4849-80b8-c5a349f19008 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:20.183472138 +0000 UTC m=+62.248968082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlnz9" (UID: "bf7c933c-1dcc-4849-80b8-c5a349f19008") : configmap "kube-rbac-proxy" not found Jul 28 17:10:19.183519 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:19.183497 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls podName:bf7c933c-1dcc-4849-80b8-c5a349f19008 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:20.183491287 +0000 UTC m=+62.248987230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlnz9" (UID: "bf7c933c-1dcc-4849-80b8-c5a349f19008") : secret "insights-runtime-extractor-tls" not found Jul 28 17:10:19.771242 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:19.771207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" event={"ID":"366e1152-89b9-42a5-b134-e23a4ffb464f","Type":"ContainerStarted","Data":"a61afb9f9d1934d3bcd1471eabf20ecfc8ba7261006ed071567f9004bc2e419e"} Jul 28 17:10:19.771680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:19.771459 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:19.776097 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:19.776072 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" Jul 28 17:10:19.791184 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:19.791139 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5xqpj" podStartSLOduration=1.401819556 podStartE2EDuration="2.791125888s" totalImagesPullingTime="1.389306332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:19.790594503 +0000 UTC m=+61.856090468" watchObservedRunningTime="2026-07-28 17:10:19.791125888 +0000 UTC m=+61.856621852" Jul 28 17:10:20.192449 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.192410 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:20.192449 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.192450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:20.192672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:20.192537 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:10:20.192672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:20.192562 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:10:20.192672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:20.192587 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm podName:bf7c933c-1dcc-4849-80b8-c5a349f19008 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.192574113 +0000 UTC m=+64.258070056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlnz9" (UID: "bf7c933c-1dcc-4849-80b8-c5a349f19008") : configmap "kube-rbac-proxy" not found Jul 28 17:10:20.192672 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:20.192612 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls podName:bf7c933c-1dcc-4849-80b8-c5a349f19008 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.192598026 +0000 UTC m=+64.258093969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlnz9" (UID: "bf7c933c-1dcc-4849-80b8-c5a349f19008") : secret "insights-runtime-extractor-tls" not found Jul 28 17:10:20.667417 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.667379 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:20.667567 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.667471 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:20.668797 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.668771 2567 patch_prober.go:28] interesting pod/console-86f454b754-6629g container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 28 17:10:20.668928 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.668810 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-86f454b754-6629g" podUID="03030ad1-2da2-49f0-85c0-93ea9ead8259" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 28 17:10:20.810842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.810810 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-78klr"] Jul 28 17:10:20.812761 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.812746 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:20.815617 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.815597 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 17:10:20.815734 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.815644 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 17:10:20.817008 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.816988 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 17:10:20.817434 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.817417 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 17:10:20.817527 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.817440 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-md4pc\"" Jul 28 17:10:20.817527 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.817448 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 17:10:20.828581 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.828433 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-78klr"] Jul 28 17:10:20.899118 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.899078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:20.899288 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.899129 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:20.899288 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.899226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmj4\" (UniqueName: \"kubernetes.io/projected/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-kube-api-access-mcmj4\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:20.899288 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.899284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:20.934518 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:20.934454 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54d99df865-jkjfs_6ddc3d4f-180d-47b9-868f-356568a5cf38/console/0.log" Jul 28 17:10:21.000087 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.000053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.000204 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.000146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmj4\" (UniqueName: \"kubernetes.io/projected/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-kube-api-access-mcmj4\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.000242 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:21.000201 2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 28 17:10:21.000275 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:21.000263 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-tls podName:4d40ac35-ed57-4d1f-83b2-b61a850e9caa nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.500247765 +0000 UTC m=+63.565743727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-78klr" (UID: "4d40ac35-ed57-4d1f-83b2-b61a850e9caa") : secret "prometheus-operator-tls" not found Jul 28 17:10:21.000317 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.000198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.000358 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.000340 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.001016 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.000993 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.002434 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.002416 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.021092 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.021066 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmj4\" (UniqueName: \"kubernetes.io/projected/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-kube-api-access-mcmj4\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.142548 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.142521 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86f454b754-6629g_03030ad1-2da2-49f0-85c0-93ea9ead8259/console/0.log" Jul 28 17:10:21.504150 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.504112 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.506405 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.506382 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d40ac35-ed57-4d1f-83b2-b61a850e9caa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-78klr\" (UID: \"4d40ac35-ed57-4d1f-83b2-b61a850e9caa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.515122 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.515089 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dp5lc_fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a/dns/0.log" Jul 28 17:10:21.716193 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.716166 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dp5lc_fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a/kube-rbac-proxy/0.log" Jul 28 17:10:21.721101 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.721080 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" Jul 28 17:10:21.850546 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:21.850515 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-78klr"] Jul 28 17:10:21.853599 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:21.853571 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d40ac35_ed57_4d1f_83b2_b61a850e9caa.slice/crio-b06e770675665ac88ad08ceb360c9deebad537cce0d00644f051fc04ddfd0827 WatchSource:0}: Error finding container b06e770675665ac88ad08ceb360c9deebad537cce0d00644f051fc04ddfd0827: Status 404 returned error can't find the container with id b06e770675665ac88ad08ceb360c9deebad537cce0d00644f051fc04ddfd0827 Jul 28 17:10:22.210343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.210311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:22.210502 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.210346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:22.210857 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.210825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf7c933c-1dcc-4849-80b8-c5a349f19008-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:22.212632 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.212611 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf7c933c-1dcc-4849-80b8-c5a349f19008-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlnz9\" (UID: \"bf7c933c-1dcc-4849-80b8-c5a349f19008\") " pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:22.429415 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.429384 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zlnz9" Jul 28 17:10:22.573089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.573053 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zlnz9"] Jul 28 17:10:22.576492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:22.576465 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7c933c_1dcc_4849_80b8_c5a349f19008.slice/crio-fac40393bf71ccd84d7b736b0ecfad58aa76689689300d07d78e41efab548297 WatchSource:0}: Error finding container fac40393bf71ccd84d7b736b0ecfad58aa76689689300d07d78e41efab548297: Status 404 returned error can't find the container with id fac40393bf71ccd84d7b736b0ecfad58aa76689689300d07d78e41efab548297 Jul 28 17:10:22.785246 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.785152 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlnz9" event={"ID":"bf7c933c-1dcc-4849-80b8-c5a349f19008","Type":"ContainerStarted","Data":"fac40393bf71ccd84d7b736b0ecfad58aa76689689300d07d78e41efab548297"} Jul 28 17:10:22.786425 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.786393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" event={"ID":"4d40ac35-ed57-4d1f-83b2-b61a850e9caa","Type":"ContainerStarted","Data":"b06e770675665ac88ad08ceb360c9deebad537cce0d00644f051fc04ddfd0827"} Jul 28 17:10:22.826897 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.826851 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:22.827054 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.826923 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:22.831893 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.831856 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:22.915123 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:22.915093 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hfc7r_aef8329d-9fa4-4901-b7b4-8dda9ee78061/dns-node-resolver/0.log" Jul 28 17:10:23.515690 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.515629 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78f6894798-z2wwk_52c988b9-2f97-47df-adf1-dda8d949890a/registry/0.log" Jul 28 17:10:23.747805 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.747774 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dp5lc" Jul 28 17:10:23.793687 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.793114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" event={"ID":"4d40ac35-ed57-4d1f-83b2-b61a850e9caa","Type":"ContainerStarted","Data":"126a3af5148511f5b4eff27fb98003d0714d28eab935a25e58d1234f2010b22d"} Jul 28 17:10:23.793687 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.793176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" event={"ID":"4d40ac35-ed57-4d1f-83b2-b61a850e9caa","Type":"ContainerStarted","Data":"2478133ebaa3b3ba5c7aaca37b11fa88db592cd4ecd0d15720ddcdbb09dff78e"} Jul 28 17:10:23.798748 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.798719 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:10:23.810711 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.810665 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-78klr" podStartSLOduration=2.647597223 podStartE2EDuration="3.810652437s" totalImagesPullingTime="1.163055214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:23.809857384 +0000 UTC m=+65.875353348" watchObservedRunningTime="2026-07-28 17:10:23.810652437 +0000 UTC m=+65.876148401" Jul 28 17:10:23.854668 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:23.854639 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f454b754-6629g"] Jul 28 17:10:24.116034 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.116007 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5g7t_00143af9-f792-4786-9bd2-aa61b5d1c32a/node-ca/0.log" Jul 28 17:10:24.127471 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.127444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:24.132200 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.132178 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:10:24.140700 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.140671 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4997a7f-0501-47fb-ad19-35c54ca8d006-original-pull-secret\") pod \"global-pull-secret-syncer-2s7mz\" (UID: \"e4997a7f-0501-47fb-ad19-35c54ca8d006\") " pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:24.228016 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.227970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:24.228177 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.228070 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:24.232160 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.232131 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:10:24.232279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.232188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:10:24.240916 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.240885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5-metrics-certs\") pod \"network-metrics-daemon-nlxkb\" (UID: \"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5\") " pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:24.241057 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.240898 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:10:24.248550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.248521 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7zr5q\"" Jul 28 17:10:24.251317 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.251153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2s7mz" Jul 28 17:10:24.251781 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.251755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mzw\" (UniqueName: \"kubernetes.io/projected/0923ada6-bef2-4608-a197-021abbcacd6d-kube-api-access-t9mzw\") pod \"network-check-target-2bfzx\" (UID: \"0923ada6-bef2-4608-a197-021abbcacd6d\") " pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:24.256283 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.256262 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlxkb" Jul 28 17:10:24.258105 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.258041 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4dt6\"" Jul 28 17:10:24.266139 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.266114 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:24.426160 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.426075 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nlxkb"] Jul 28 17:10:24.430843 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:24.430806 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfc9106_ddc9_4e2c_bdfe_a89314bf9fd5.slice/crio-ea080091210a62d01c75b61aafd8ec5f39dc7a31e9bbaac079a8c8789441abf0 WatchSource:0}: Error finding container ea080091210a62d01c75b61aafd8ec5f39dc7a31e9bbaac079a8c8789441abf0: Status 404 returned error can't find the container with id ea080091210a62d01c75b61aafd8ec5f39dc7a31e9bbaac079a8c8789441abf0 Jul 28 17:10:24.652858 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.652779 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2s7mz"] Jul 28 17:10:24.655373 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.655348 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2bfzx"] Jul 28 17:10:24.656454 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:24.656402 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4997a7f_0501_47fb_ad19_35c54ca8d006.slice/crio-fe401e3d4ff345bc5a5969878d1eef254bf462414d9e1b30f0e2469917e1436d WatchSource:0}: Error finding container fe401e3d4ff345bc5a5969878d1eef254bf462414d9e1b30f0e2469917e1436d: Status 404 returned error can't find the container with id fe401e3d4ff345bc5a5969878d1eef254bf462414d9e1b30f0e2469917e1436d Jul 28 17:10:24.658874 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:24.658835 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0923ada6_bef2_4608_a197_021abbcacd6d.slice/crio-11a1725f7b475f1e3fae50e4b47cd692256618a951e8db933509410417832350 WatchSource:0}: Error finding container 11a1725f7b475f1e3fae50e4b47cd692256618a951e8db933509410417832350: Status 404 returned error can't find the container with id 11a1725f7b475f1e3fae50e4b47cd692256618a951e8db933509410417832350 Jul 28 17:10:24.714981 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.714955 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-622rh_5bb2f701-67fa-48f1-9afd-0c7ecd04039a/serve-healthcheck-canary/0.log" Jul 28 17:10:24.798315 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.798273 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2bfzx" event={"ID":"0923ada6-bef2-4608-a197-021abbcacd6d","Type":"ContainerStarted","Data":"11a1725f7b475f1e3fae50e4b47cd692256618a951e8db933509410417832350"} Jul 28 17:10:24.799601 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.799567 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2s7mz" event={"ID":"e4997a7f-0501-47fb-ad19-35c54ca8d006","Type":"ContainerStarted","Data":"fe401e3d4ff345bc5a5969878d1eef254bf462414d9e1b30f0e2469917e1436d"} Jul 28 17:10:24.800987 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:24.800956 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlxkb" event={"ID":"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5","Type":"ContainerStarted","Data":"ea080091210a62d01c75b61aafd8ec5f39dc7a31e9bbaac079a8c8789441abf0"} Jul 28 17:10:26.185915 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.185884 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8"] Jul 28 17:10:26.205553 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.205526 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.208927 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.208635 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 17:10:26.210310 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.210073 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:10:26.210310 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.210094 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6pm5b\"" Jul 28 17:10:26.210541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.210507 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8"] Jul 28 17:10:26.211784 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.211756 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k"] Jul 28 17:10:26.216716 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.216698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.219612 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.219581 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 17:10:26.219718 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.219624 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 17:10:26.219718 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.219674 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qcq6d\"" Jul 28 17:10:26.219940 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.219923 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:10:26.222617 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.222598 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wcwdt"] Jul 28 17:10:26.226271 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.226251 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k"] Jul 28 17:10:26.226367 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.226318 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.229283 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.229233 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 17:10:26.229504 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.229485 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 17:10:26.229805 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.229787 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 17:10:26.229805 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.229798 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvlvp\"" Jul 28 17:10:26.246856 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.246831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.246977 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.246878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsm2\" (UniqueName: \"kubernetes.io/projected/5af600dd-57af-45ad-9b7f-9bca4a252792-kube-api-access-hvsm2\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.246977 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.246911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9560b20a-d829-49f2-a693-f56c76a2297a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.246977 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.246948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5af600dd-57af-45ad-9b7f-9bca4a252792-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.247089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.247014 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.247089 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.247081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9560b20a-d829-49f2-a693-f56c76a2297a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.247183 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.247109 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.247183 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.247142 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mpk\" (UniqueName: \"kubernetes.io/projected/9560b20a-d829-49f2-a693-f56c76a2297a-kube-api-access-h4mpk\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.247183 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.247175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.247318 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.247195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.347928 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.347893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9560b20a-d829-49f2-a693-f56c76a2297a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348106 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.347934 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.348106 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.347968 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-textfile\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348106 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mpk\" (UniqueName: \"kubernetes.io/projected/9560b20a-d829-49f2-a693-f56c76a2297a-kube-api-access-h4mpk\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348106 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-root\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348106 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:26.348094 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-metrics-client-ca\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:26.348170 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-tls podName:5af600dd-57af-45ad-9b7f-9bca4a252792 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.848146533 +0000 UTC m=+68.913642478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-9zlg8" (UID: "5af600dd-57af-45ad-9b7f-9bca4a252792") : secret "openshift-state-metrics-tls" not found Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348257 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmln9\" (UniqueName: \"kubernetes.io/projected/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-kube-api-access-qmln9\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348337 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:26.348328 2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:26.348394 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-tls podName:9560b20a-d829-49f2-a693-f56c76a2297a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.848374903 +0000 UTC m=+68.913870846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-s7x2k" (UID: "9560b20a-d829-49f2-a693-f56c76a2297a") : secret "kube-state-metrics-tls" not found Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9560b20a-d829-49f2-a693-f56c76a2297a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-sys\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsm2\" (UniqueName: \"kubernetes.io/projected/5af600dd-57af-45ad-9b7f-9bca4a252792-kube-api-access-hvsm2\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-tls\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9560b20a-d829-49f2-a693-f56c76a2297a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5af600dd-57af-45ad-9b7f-9bca4a252792-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348593 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.348679 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.348618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-wtmp\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.349088 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.349064 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.349324 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.349300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5af600dd-57af-45ad-9b7f-9bca4a252792-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.349435 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.349378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9560b20a-d829-49f2-a693-f56c76a2297a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.351273 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.351247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.351382 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.351312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.357492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.357448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsm2\" (UniqueName: \"kubernetes.io/projected/5af600dd-57af-45ad-9b7f-9bca4a252792-kube-api-access-hvsm2\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.358528 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.358496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mpk\" (UniqueName: \"kubernetes.io/projected/9560b20a-d829-49f2-a693-f56c76a2297a-kube-api-access-h4mpk\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.449343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.449242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-sys\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.449343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.449295 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-tls\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.449343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.449340 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-wtmp\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.449650 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.449375 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-sys\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.449744 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.449414 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-textfile\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.449799 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.449786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-root\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:26.450111 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450140 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-wtmp\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:10:26.450202 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-tls podName:a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.950172129 +0000 UTC m=+69.015668078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-tls") pod "node-exporter-wcwdt" (UID: "a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5") : secret "node-exporter-tls" not found Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-root\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450275 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-metrics-client-ca\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.450480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-textfile\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.451037 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.450988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-metrics-client-ca\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.451129 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.451006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmln9\" (UniqueName: \"kubernetes.io/projected/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-kube-api-access-qmln9\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.452881 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.451390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.454399 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.453350 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.463028 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.463003 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmln9\" (UniqueName: \"kubernetes.io/projected/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-kube-api-access-qmln9\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.855736 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.855700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.856069 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.855818 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.858443 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.858413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5af600dd-57af-45ad-9b7f-9bca4a252792-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-9zlg8\" (UID: \"5af600dd-57af-45ad-9b7f-9bca4a252792\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:26.858550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.858501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9560b20a-d829-49f2-a693-f56c76a2297a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-s7x2k\" (UID: \"9560b20a-d829-49f2-a693-f56c76a2297a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:26.956763 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.956726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-tls\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:26.959364 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:26.959342 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5-node-exporter-tls\") pod \"node-exporter-wcwdt\" (UID: \"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5\") " pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:27.117970 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.117897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" Jul 28 17:10:27.131121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.131082 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" Jul 28 17:10:27.138315 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.138245 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wcwdt" Jul 28 17:10:27.327997 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.327962 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:10:27.350547 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.350518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.351674 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.351383 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:10:27.361499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.361469 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.361855 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.361968 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.362069 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.362141 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.362229 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.361855 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.362402 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.362456 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kxn59\"" Jul 28 17:10:27.362550 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.362563 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 17:10:27.464178 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464196 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7acbbb3-8e15-47db-8361-31f545deba15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464237 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-web-config\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464314 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464466 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464342 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7acbbb3-8e15-47db-8361-31f545deba15-config-out\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464466 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464368 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464466 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65ms\" (UniqueName: \"kubernetes.io/projected/a7acbbb3-8e15-47db-8361-31f545deba15-kube-api-access-k65ms\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464466 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464602 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7acbbb3-8e15-47db-8361-31f545deba15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464602 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7acbbb3-8e15-47db-8361-31f545deba15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464602 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.464602 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.464560 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7acbbb3-8e15-47db-8361-31f545deba15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566094 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.565794 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-web-config\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566094 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.565883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566094 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.565916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7acbbb3-8e15-47db-8361-31f545deba15-config-out\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566094 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.565938 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566413 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566228 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65ms\" (UniqueName: \"kubernetes.io/projected/a7acbbb3-8e15-47db-8361-31f545deba15-kube-api-access-k65ms\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566413 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566269 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566413 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7acbbb3-8e15-47db-8361-31f545deba15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566413 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566361 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7acbbb3-8e15-47db-8361-31f545deba15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566413 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7acbbb3-8e15-47db-8361-31f545deba15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7acbbb3-8e15-47db-8361-31f545deba15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.566657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.566532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.567433 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.567401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7acbbb3-8e15-47db-8361-31f545deba15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.567614 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.567592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7acbbb3-8e15-47db-8361-31f545deba15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.567888 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.567844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a7acbbb3-8e15-47db-8361-31f545deba15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.571686 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.571663 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.572138 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.572116 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.572333 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.572268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a7acbbb3-8e15-47db-8361-31f545deba15-config-out\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.572757 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.572738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-web-config\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.573501 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.573456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.574583 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.574543 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.575461 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.575440 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.576987 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.576963 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a7acbbb3-8e15-47db-8361-31f545deba15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.577819 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.577789 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65ms\" (UniqueName: \"kubernetes.io/projected/a7acbbb3-8e15-47db-8361-31f545deba15-kube-api-access-k65ms\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.578418 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.578395 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a7acbbb3-8e15-47db-8361-31f545deba15-config-volume\") pod \"alertmanager-main-0\" (UID: \"a7acbbb3-8e15-47db-8361-31f545deba15\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:27.672675 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:27.672577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:10:28.301901 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.300602 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7"] Jul 28 17:10:28.327946 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.326310 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7"] Jul 28 17:10:28.327946 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.326466 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.331220 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.331194 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 17:10:28.331914 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.331419 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nsqg6\"" Jul 28 17:10:28.331914 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.331620 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 17:10:28.331914 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.331802 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 17:10:28.333206 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.332581 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 17:10:28.333206 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.332812 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dt4oupon2hohm\"" Jul 28 17:10:28.333206 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.333030 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 17:10:28.373343 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373307 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373495 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4fcc8e-4446-47d8-8676-37679d510c18-metrics-client-ca\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373495 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373495 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373667 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-tls\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373667 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373631 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373772 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nwc\" (UniqueName: \"kubernetes.io/projected/ea4fcc8e-4446-47d8-8676-37679d510c18-kube-api-access-h9nwc\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.373772 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.373729 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-grpc-tls\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474346 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474422 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-tls\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474503 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nwc\" (UniqueName: \"kubernetes.io/projected/ea4fcc8e-4446-47d8-8676-37679d510c18-kube-api-access-h9nwc\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-grpc-tls\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474796 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.474796 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.474599 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4fcc8e-4446-47d8-8676-37679d510c18-metrics-client-ca\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.476258 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.475510 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4fcc8e-4446-47d8-8676-37679d510c18-metrics-client-ca\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.479188 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.479130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.479740 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.479718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-tls\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.480082 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.480060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-grpc-tls\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.481009 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.480984 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.482411 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.482386 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.484188 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.484138 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ea4fcc8e-4446-47d8-8676-37679d510c18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.485857 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.485806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nwc\" (UniqueName: \"kubernetes.io/projected/ea4fcc8e-4446-47d8-8676-37679d510c18-kube-api-access-h9nwc\") pod \"thanos-querier-5d56d77cf6-wgvb7\" (UID: \"ea4fcc8e-4446-47d8-8676-37679d510c18\") " pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.641820 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.641766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:28.646895 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:28.646838 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e59893_ed27_4dcc_87ec_eb5e46d9e8b5.slice/crio-fd0d718747f17f4bec0882aaabd399ad3f960d9bcee068331d2febd8b0e8650f WatchSource:0}: Error finding container fd0d718747f17f4bec0882aaabd399ad3f960d9bcee068331d2febd8b0e8650f: Status 404 returned error can't find the container with id fd0d718747f17f4bec0882aaabd399ad3f960d9bcee068331d2febd8b0e8650f Jul 28 17:10:28.828063 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.827985 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wcwdt" event={"ID":"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5","Type":"ContainerStarted","Data":"fd0d718747f17f4bec0882aaabd399ad3f960d9bcee068331d2febd8b0e8650f"} Jul 28 17:10:28.899898 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.899725 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k"] Jul 28 17:10:28.906977 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:28.906943 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9560b20a_d829_49f2_a693_f56c76a2297a.slice/crio-b37d8518b54ab969ce32aecddfc59451c3a992197209e7ed6b9a6b064ecf85a5 WatchSource:0}: Error finding container b37d8518b54ab969ce32aecddfc59451c3a992197209e7ed6b9a6b064ecf85a5: Status 404 returned error can't find the container with id b37d8518b54ab969ce32aecddfc59451c3a992197209e7ed6b9a6b064ecf85a5 Jul 28 17:10:28.939437 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:28.939395 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:10:29.139594 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.139514 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8"] Jul 28 17:10:29.143621 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:29.143580 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af600dd_57af_45ad_9b7f_9bca4a252792.slice/crio-bd19655f1b8fd549fa8268e7c0c7495d01070dffcbfdc53f36e3b802e39b36b6 WatchSource:0}: Error finding container bd19655f1b8fd549fa8268e7c0c7495d01070dffcbfdc53f36e3b802e39b36b6: Status 404 returned error can't find the container with id bd19655f1b8fd549fa8268e7c0c7495d01070dffcbfdc53f36e3b802e39b36b6 Jul 28 17:10:29.151492 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.151460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7"] Jul 28 17:10:29.833832 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.833783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlnz9" event={"ID":"bf7c933c-1dcc-4849-80b8-c5a349f19008","Type":"ContainerStarted","Data":"4a3a2fdd5e5f5d6cf6e9e05fd726193a9456831c23c869074e6958ca723442ae"} Jul 28 17:10:29.836215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.836125 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlxkb" event={"ID":"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5","Type":"ContainerStarted","Data":"42dd9dafaede052d582580990aeedbb3988ee3c83480d4062e42f23ca122dd59"} Jul 28 17:10:29.836215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.836177 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlxkb" event={"ID":"acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5","Type":"ContainerStarted","Data":"301e0e4b2cafb810582186c44fffed9f928a45d6b1b01a68171998339c1c5936"} Jul 28 17:10:29.838595 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.838562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"873ef023c15fd576644f858a5ea7838c1e79b62472ed7ddefbdaca488650f174"} Jul 28 17:10:29.841226 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.841108 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" event={"ID":"5af600dd-57af-45ad-9b7f-9bca4a252792","Type":"ContainerStarted","Data":"2068f00e029b75b3f73097fe9ab38118e0314cd318e6fa2ea1c3ce643e11c663"} Jul 28 17:10:29.841226 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.841162 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" event={"ID":"5af600dd-57af-45ad-9b7f-9bca4a252792","Type":"ContainerStarted","Data":"cbd10b94dc37166a49800756465af5ad0a85621896f3bfd8eb2003b48d667b69"} Jul 28 17:10:29.841226 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.841191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" event={"ID":"5af600dd-57af-45ad-9b7f-9bca4a252792","Type":"ContainerStarted","Data":"bd19655f1b8fd549fa8268e7c0c7495d01070dffcbfdc53f36e3b802e39b36b6"} Jul 28 17:10:29.842333 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.842308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"73434ad5290de94be673ef264463e2f01c0ad078b579a444ea6c53a7d6536c56"} Jul 28 17:10:29.843518 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.843492 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" event={"ID":"9560b20a-d829-49f2-a693-f56c76a2297a","Type":"ContainerStarted","Data":"b37d8518b54ab969ce32aecddfc59451c3a992197209e7ed6b9a6b064ecf85a5"} Jul 28 17:10:29.854946 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:29.854904 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nlxkb" podStartSLOduration=66.64569301 podStartE2EDuration="1m10.854888337s" totalImagesPullingTime="4.209195327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:29.854583268 +0000 UTC m=+71.920079242" watchObservedRunningTime="2026-07-28 17:10:29.854888337 +0000 UTC m=+71.920384296" Jul 28 17:10:30.542904 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.542384 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77b78d6768-nbdds"] Jul 28 17:10:30.546580 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.546557 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.551965 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.551556 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 17:10:30.551965 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.551820 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 17:10:30.552839 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.552563 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 17:10:30.552839 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.552675 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 17:10:30.552839 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.552722 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cgthl\"" Jul 28 17:10:30.552839 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.552761 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-18v70fkop7tue\"" Jul 28 17:10:30.560449 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.560352 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77b78d6768-nbdds"] Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-secret-metrics-server-client-certs\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-secret-metrics-server-tls\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3387c220-7739-484c-abb4-32355494331f-audit-log\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnj6\" (UniqueName: \"kubernetes.io/projected/3387c220-7739-484c-abb4-32355494331f-kube-api-access-blnj6\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3387c220-7739-484c-abb4-32355494331f-metrics-server-audit-profiles\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595891 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3387c220-7739-484c-abb4-32355494331f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.596012 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.595906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-client-ca-bundle\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697056 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-secret-metrics-server-tls\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697239 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697081 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3387c220-7739-484c-abb4-32355494331f-audit-log\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697239 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnj6\" (UniqueName: \"kubernetes.io/projected/3387c220-7739-484c-abb4-32355494331f-kube-api-access-blnj6\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697239 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3387c220-7739-484c-abb4-32355494331f-metrics-server-audit-profiles\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697239 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3387c220-7739-484c-abb4-32355494331f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697239 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-client-ca-bundle\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.697500 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.697292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-secret-metrics-server-client-certs\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.698435 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.698403 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3387c220-7739-484c-abb4-32355494331f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.699336 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.699310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3387c220-7739-484c-abb4-32355494331f-metrics-server-audit-profiles\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.699994 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.699966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3387c220-7739-484c-abb4-32355494331f-audit-log\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.705130 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.705081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-secret-metrics-server-client-certs\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.705685 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.705595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-secret-metrics-server-tls\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.708191 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.708135 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnj6\" (UniqueName: \"kubernetes.io/projected/3387c220-7739-484c-abb4-32355494331f-kube-api-access-blnj6\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.709535 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.709456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3387c220-7739-484c-abb4-32355494331f-client-ca-bundle\") pod \"metrics-server-77b78d6768-nbdds\" (UID: \"3387c220-7739-484c-abb4-32355494331f\") " pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:30.861454 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:30.861412 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:31.000335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.000298 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k"] Jul 28 17:10:31.003800 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.003776 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.007228 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.007060 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wp2vx\"" Jul 28 17:10:31.007228 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.007061 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 17:10:31.027589 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.027559 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k"] Jul 28 17:10:31.101721 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.101660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93d4bef0-9df1-49ae-9352-59c92ea90981-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m8q2k\" (UID: \"93d4bef0-9df1-49ae-9352-59c92ea90981\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.101910 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.101744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvx2\" (UniqueName: \"kubernetes.io/projected/93d4bef0-9df1-49ae-9352-59c92ea90981-kube-api-access-8wvx2\") pod \"monitoring-plugin-68b7fc9574-m8q2k\" (UID: \"93d4bef0-9df1-49ae-9352-59c92ea90981\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.202567 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.202479 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93d4bef0-9df1-49ae-9352-59c92ea90981-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m8q2k\" (UID: \"93d4bef0-9df1-49ae-9352-59c92ea90981\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.202749 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.202581 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvx2\" (UniqueName: \"kubernetes.io/projected/93d4bef0-9df1-49ae-9352-59c92ea90981-kube-api-access-8wvx2\") pod \"monitoring-plugin-68b7fc9574-m8q2k\" (UID: \"93d4bef0-9df1-49ae-9352-59c92ea90981\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.214559 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.214529 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/93d4bef0-9df1-49ae-9352-59c92ea90981-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-m8q2k\" (UID: \"93d4bef0-9df1-49ae-9352-59c92ea90981\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.217611 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.217583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvx2\" (UniqueName: \"kubernetes.io/projected/93d4bef0-9df1-49ae-9352-59c92ea90981-kube-api-access-8wvx2\") pod \"monitoring-plugin-68b7fc9574-m8q2k\" (UID: \"93d4bef0-9df1-49ae-9352-59c92ea90981\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.320276 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.319968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:31.486017 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.485136 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n"] Jul 28 17:10:31.489231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.489208 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.494440 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.494417 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 17:10:31.494564 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.494462 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 17:10:31.494564 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.494528 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qhgzj\"" Jul 28 17:10:31.494720 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.494693 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 17:10:31.494766 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.494698 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 17:10:31.494766 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.494757 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 17:10:31.500807 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.500783 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 17:10:31.506531 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.506506 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n"] Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-federate-client-tls\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-serving-certs-ca-bundle\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-telemeter-client-tls\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606402 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-secret-telemeter-client\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jnf\" (UniqueName: \"kubernetes.io/projected/eb004635-2806-43ea-af5b-28d931fe356f-kube-api-access-j7jnf\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606529 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.606682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.606556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-metrics-client-ca\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707412 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-secret-telemeter-client\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707584 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707584 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jnf\" (UniqueName: \"kubernetes.io/projected/eb004635-2806-43ea-af5b-28d931fe356f-kube-api-access-j7jnf\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707584 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707584 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-metrics-client-ca\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-federate-client-tls\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-serving-certs-ca-bundle\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.707790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.707690 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-telemeter-client-tls\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.708992 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.708612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-serving-certs-ca-bundle\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.708992 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.708826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.708992 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.708941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb004635-2806-43ea-af5b-28d931fe356f-metrics-client-ca\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.710726 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.710707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-telemeter-client-tls\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.711044 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.711024 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-federate-client-tls\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.711128 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.711025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.711569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.711548 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb004635-2806-43ea-af5b-28d931fe356f-secret-telemeter-client\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.718422 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.718378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jnf\" (UniqueName: \"kubernetes.io/projected/eb004635-2806-43ea-af5b-28d931fe356f-kube-api-access-j7jnf\") pod \"telemeter-client-5dfff7fddf-gdf6n\" (UID: \"eb004635-2806-43ea-af5b-28d931fe356f\") " pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:31.736826 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.736765 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78f6894798-z2wwk" Jul 28 17:10:31.804963 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:31.804930 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" Jul 28 17:10:32.482888 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.482457 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:10:32.487572 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.487552 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.490463 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.490437 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 17:10:32.490565 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.490538 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 17:10:32.490843 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.490825 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 17:10:32.490843 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.490835 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 17:10:32.491011 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.490833 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 17:10:32.491851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.491711 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 17:10:32.491851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.491745 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s5qv9\"" Jul 28 17:10:32.491851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.491823 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 17:10:32.491851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.491823 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 17:10:32.491851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.491842 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 17:10:32.491851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.491836 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 17:10:32.492198 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.492039 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 17:10:32.492198 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.492083 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8sjuk0cook6r8\"" Jul 28 17:10:32.492361 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.492337 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 17:10:32.495531 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.495511 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 17:10:32.499363 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.498949 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 17:10:32.500311 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.500287 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617425 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617484 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617545 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-config\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617654 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617687 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s792\" (UniqueName: \"kubernetes.io/projected/55c73256-263d-47f0-8f4b-dcdd4049e568-kube-api-access-8s792\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617745 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617797 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55c73256-263d-47f0-8f4b-dcdd4049e568-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.617920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617894 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.618755 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.617946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.618755 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.618043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55c73256-263d-47f0-8f4b-dcdd4049e568-config-out\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.618755 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.618088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.618755 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.618125 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-web-config\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.618755 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.618180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.718891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55c73256-263d-47f0-8f4b-dcdd4049e568-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.718943 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.718974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719002 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719078 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55c73256-263d-47f0-8f4b-dcdd4049e568-config-out\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-web-config\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719193 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-config\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.721313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719433 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.722325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.722325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s792\" (UniqueName: \"kubernetes.io/projected/55c73256-263d-47f0-8f4b-dcdd4049e568-kube-api-access-8s792\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.722325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.719531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.722325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.720955 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.722325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.721612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.727049 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.723206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.727049 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.723995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.727049 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.724816 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.727049 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.725450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.730655 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.728957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55c73256-263d-47f0-8f4b-dcdd4049e568-config-out\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.730655 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.729650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.730655 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.730237 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/55c73256-263d-47f0-8f4b-dcdd4049e568-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.735917 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.732593 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-config\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.735917 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.733087 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f696545-fcmhv"] Jul 28 17:10:32.735917 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.733627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55c73256-263d-47f0-8f4b-dcdd4049e568-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.735917 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.734084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-web-config\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.735917 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.734450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.735917 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.735141 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.738532 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.738443 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.739946 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.739274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.740210 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.740154 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.741147 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.741121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s792\" (UniqueName: \"kubernetes.io/projected/55c73256-263d-47f0-8f4b-dcdd4049e568-kube-api-access-8s792\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.741746 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.741700 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.742306 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.742267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/55c73256-263d-47f0-8f4b-dcdd4049e568-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"55c73256-263d-47f0-8f4b-dcdd4049e568\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.745555 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.743681 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.753011 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.752989 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f696545-fcmhv"] Jul 28 17:10:32.767196 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.767171 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77b78d6768-nbdds"] Jul 28 17:10:32.802561 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.802336 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:32.822947 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.822908 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-service-ca\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.823133 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.822962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-oauth-config\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.823133 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.823009 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-config\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.823133 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.823036 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-trusted-ca-bundle\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.823133 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.823082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j7s\" (UniqueName: \"kubernetes.io/projected/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-kube-api-access-g7j7s\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.823364 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.823173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-serving-cert\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.823364 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.823198 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-oauth-serving-cert\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.926906 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.926823 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k"] Jul 28 17:10:32.928796 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.928707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-service-ca\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.928796 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.928748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-oauth-config\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.928796 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.928784 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-config\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.929014 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.928814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-trusted-ca-bundle\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.928857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j7s\" (UniqueName: \"kubernetes.io/projected/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-kube-api-access-g7j7s\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.929826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-serving-cert\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.929837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-config\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.929858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-oauth-serving-cert\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.930072 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-service-ca\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.930801 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-trusted-ca-bundle\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931251 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.931217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-oauth-serving-cert\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.931730 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.931701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n"] Jul 28 17:10:32.939455 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.939408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-oauth-config\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.939582 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.939559 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j7s\" (UniqueName: \"kubernetes.io/projected/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-kube-api-access-g7j7s\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:32.940366 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:32.940340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-serving-cert\") pod \"console-56f696545-fcmhv\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:33.079891 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:33.079794 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:33.080648 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:33.080538 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3387c220_7739_484c_abb4_32355494331f.slice/crio-559a9dc9149b9d376948dee49715356ab274ad5059504835e8f62209643bc5f6 WatchSource:0}: Error finding container 559a9dc9149b9d376948dee49715356ab274ad5059504835e8f62209643bc5f6: Status 404 returned error can't find the container with id 559a9dc9149b9d376948dee49715356ab274ad5059504835e8f62209643bc5f6 Jul 28 17:10:33.869108 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:33.869060 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" event={"ID":"3387c220-7739-484c-abb4-32355494331f","Type":"ContainerStarted","Data":"559a9dc9149b9d376948dee49715356ab274ad5059504835e8f62209643bc5f6"} Jul 28 17:10:33.870757 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:33.870721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2bfzx" event={"ID":"0923ada6-bef2-4608-a197-021abbcacd6d","Type":"ContainerStarted","Data":"6352b7711022a1115d9e397b97c873fe648d8e88d154dd24865c6e58beb66f3f"} Jul 28 17:10:33.870900 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:33.870851 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:10:33.890625 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:33.890576 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2bfzx" podStartSLOduration=68.031583194 podStartE2EDuration="1m15.890561s" totalImagesPullingTime="7.858977806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:33.88873605 +0000 UTC m=+75.954232030" watchObservedRunningTime="2026-07-28 17:10:33.890561 +0000 UTC m=+75.956056965" Jul 28 17:10:34.967558 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:34.967526 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d4bef0_9df1_49ae_9352_59c92ea90981.slice/crio-239aa65ba70e3bd7cacb8f3e7ae72d940af0f358dc6b8edc8e372ebf8a032c24 WatchSource:0}: Error finding container 239aa65ba70e3bd7cacb8f3e7ae72d940af0f358dc6b8edc8e372ebf8a032c24: Status 404 returned error can't find the container with id 239aa65ba70e3bd7cacb8f3e7ae72d940af0f358dc6b8edc8e372ebf8a032c24 Jul 28 17:10:35.203319 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:35.203286 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb004635_2806_43ea_af5b_28d931fe356f.slice/crio-4a4219470e17834b357ce888bdf3b43f67b8a6da6002145a4b78793247ead38d WatchSource:0}: Error finding container 4a4219470e17834b357ce888bdf3b43f67b8a6da6002145a4b78793247ead38d: Status 404 returned error can't find the container with id 4a4219470e17834b357ce888bdf3b43f67b8a6da6002145a4b78793247ead38d Jul 28 17:10:35.881983 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:35.881895 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wcwdt" event={"ID":"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5","Type":"ContainerStarted","Data":"837a00d737e5372e67feb7aa2ab0ede6b71fc362a0387b19229858a9969c4301"} Jul 28 17:10:35.883509 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:35.883465 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" event={"ID":"93d4bef0-9df1-49ae-9352-59c92ea90981","Type":"ContainerStarted","Data":"239aa65ba70e3bd7cacb8f3e7ae72d940af0f358dc6b8edc8e372ebf8a032c24"} Jul 28 17:10:35.884836 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:35.884809 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" event={"ID":"eb004635-2806-43ea-af5b-28d931fe356f","Type":"ContainerStarted","Data":"4a4219470e17834b357ce888bdf3b43f67b8a6da6002145a4b78793247ead38d"} Jul 28 17:10:36.891351 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:36.891314 2567 generic.go:356] "Generic (PLEG): container finished" podID="a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5" containerID="837a00d737e5372e67feb7aa2ab0ede6b71fc362a0387b19229858a9969c4301" exitCode=0 Jul 28 17:10:36.891721 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:36.891421 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wcwdt" event={"ID":"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5","Type":"ContainerDied","Data":"837a00d737e5372e67feb7aa2ab0ede6b71fc362a0387b19229858a9969c4301"} Jul 28 17:10:37.487153 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.487112 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f696545-fcmhv"] Jul 28 17:10:37.500093 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:37.500050 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59b27f6_a2b3_4dcb_b68c_1efa5325269e.slice/crio-82927a52e2383e9c43148330fa7de9dc5ff3c7fe103e9b3a9f4e80d285f6627f WatchSource:0}: Error finding container 82927a52e2383e9c43148330fa7de9dc5ff3c7fe103e9b3a9f4e80d285f6627f: Status 404 returned error can't find the container with id 82927a52e2383e9c43148330fa7de9dc5ff3c7fe103e9b3a9f4e80d285f6627f Jul 28 17:10:37.526739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.526474 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 17:10:37.903189 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.902677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" event={"ID":"93d4bef0-9df1-49ae-9352-59c92ea90981","Type":"ContainerStarted","Data":"2b0de9ca002f1a7ba481188c00572a24a430cb11d496647cb5ca3b9773a962d2"} Jul 28 17:10:37.903189 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.902745 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:37.904851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.904814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2s7mz" event={"ID":"e4997a7f-0501-47fb-ad19-35c54ca8d006","Type":"ContainerStarted","Data":"4d95fc097242dfe3add1619d69c7cc07ef7e6655e73780ef2c053cf09f51d47d"} Jul 28 17:10:37.907192 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.907113 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"c9f224921ea4700fde697fee69160066ab3b08362895cc2dcc9b3af6f2d382d1"} Jul 28 17:10:37.910821 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.910771 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" Jul 28 17:10:37.912293 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.912218 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlnz9" event={"ID":"bf7c933c-1dcc-4849-80b8-c5a349f19008","Type":"ContainerStarted","Data":"fac0beb5b775514c584bda218e371e1accf12f3cfcd9a2d13740dc06185f0e87"} Jul 28 17:10:37.914097 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.914070 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"6a7e946859dcd356101caca405636f8d1271e40dc5e1619f7d9a491d03c137cf"} Jul 28 17:10:37.915558 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.915535 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f696545-fcmhv" event={"ID":"e59b27f6-a2b3-4dcb-b68c-1efa5325269e","Type":"ContainerStarted","Data":"82927a52e2383e9c43148330fa7de9dc5ff3c7fe103e9b3a9f4e80d285f6627f"} Jul 28 17:10:37.917930 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.917856 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wcwdt" event={"ID":"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5","Type":"ContainerStarted","Data":"0f76a1b363b3451bcfe2d51f33340182b58b643b796a711ce15821fca3b628a4"} Jul 28 17:10:37.921750 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.921678 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" event={"ID":"5af600dd-57af-45ad-9b7f-9bca4a252792","Type":"ContainerStarted","Data":"7f0f6ab01a0853f37d929db2f6029d28b291a7fd4f636e5ceee05e8ef4f2c03f"} Jul 28 17:10:37.923753 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.923729 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" event={"ID":"9560b20a-d829-49f2-a693-f56c76a2297a","Type":"ContainerStarted","Data":"8126e271c5a7580810f419035b43fce0bfb81243b1a516e3a789a30d7274db1b"} Jul 28 17:10:37.924455 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.924404 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-m8q2k" podStartSLOduration=5.536193129 podStartE2EDuration="7.924384795s" totalImagesPullingTime="2.388191666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:37.92255721 +0000 UTC m=+79.988053176" watchObservedRunningTime="2026-07-28 17:10:37.924384795 +0000 UTC m=+79.989880755" Jul 28 17:10:37.925654 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.925630 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" event={"ID":"3387c220-7739-484c-abb4-32355494331f","Type":"ContainerStarted","Data":"9f5c5f47c73d16c9c6d4aa4c9af31b3c9172e0d71641fe6a5a5cc038b0b814b8"} Jul 28 17:10:37.947675 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:37.947620 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-9zlg8" podStartSLOduration=4.732972951 podStartE2EDuration="11.947599269s" totalImagesPullingTime="7.214626318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:37.945615506 +0000 UTC m=+80.011111471" watchObservedRunningTime="2026-07-28 17:10:37.947599269 +0000 UTC m=+80.013095237" Jul 28 17:10:38.054444 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.054381 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2s7mz" podStartSLOduration=71.17886747 podStartE2EDuration="1m19.054359002s" totalImagesPullingTime="7.875491532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.053257063 +0000 UTC m=+80.118753029" watchObservedRunningTime="2026-07-28 17:10:38.054359002 +0000 UTC m=+80.119854951" Jul 28 17:10:38.092238 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.092181 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" podStartSLOduration=3.817061477 podStartE2EDuration="8.092159729s" totalImagesPullingTime="4.275098252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.089820325 +0000 UTC m=+80.155316270" watchObservedRunningTime="2026-07-28 17:10:38.092159729 +0000 UTC m=+80.157655695" Jul 28 17:10:38.247709 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:38.247679 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c73256_263d_47f0_8f4b_dcdd4049e568.slice/crio-530f3dcf4e91ee1eeea384a9e5eda9c1cb6e1d7057f7885562d2704efff3753b WatchSource:0}: Error finding container 530f3dcf4e91ee1eeea384a9e5eda9c1cb6e1d7057f7885562d2704efff3753b: Status 404 returned error can't find the container with id 530f3dcf4e91ee1eeea384a9e5eda9c1cb6e1d7057f7885562d2704efff3753b Jul 28 17:10:38.933424 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.933380 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" event={"ID":"9560b20a-d829-49f2-a693-f56c76a2297a","Type":"ContainerStarted","Data":"9e5baa160e2a6405627e81908c83d97941e873a96b5511fd08f3a976ffcaef54"} Jul 28 17:10:38.933828 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.933448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" event={"ID":"9560b20a-d829-49f2-a693-f56c76a2297a","Type":"ContainerStarted","Data":"c4fed00871f78165615f23cfa0692e87ad7225a551e85ac5a44c34dd237ffd8c"} Jul 28 17:10:38.935920 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.935887 2567 generic.go:356] "Generic (PLEG): container finished" podID="55c73256-263d-47f0-8f4b-dcdd4049e568" containerID="6e711f89b77b348195d3c5cbaf5100b27dcb97b25a73ebf260e2636381ee4f93" exitCode=0 Jul 28 17:10:38.936044 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.935987 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerDied","Data":"6e711f89b77b348195d3c5cbaf5100b27dcb97b25a73ebf260e2636381ee4f93"} Jul 28 17:10:38.936127 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.936061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"530f3dcf4e91ee1eeea384a9e5eda9c1cb6e1d7057f7885562d2704efff3753b"} Jul 28 17:10:38.938408 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.938389 2567 generic.go:356] "Generic (PLEG): container finished" podID="a7acbbb3-8e15-47db-8361-31f545deba15" containerID="c9f224921ea4700fde697fee69160066ab3b08362895cc2dcc9b3af6f2d382d1" exitCode=0 Jul 28 17:10:38.938481 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.938458 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerDied","Data":"c9f224921ea4700fde697fee69160066ab3b08362895cc2dcc9b3af6f2d382d1"} Jul 28 17:10:38.942626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.942503 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" event={"ID":"eb004635-2806-43ea-af5b-28d931fe356f","Type":"ContainerStarted","Data":"ca720e1e937d97473c2f4bdf1150c9039c93be7b8c9bb4950e15c6ae2faf3864"} Jul 28 17:10:38.942626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.942586 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" event={"ID":"eb004635-2806-43ea-af5b-28d931fe356f","Type":"ContainerStarted","Data":"4f0a4af712854e1fba609df1e84fc55776bf1d34affb694768c627ace80f9d7b"} Jul 28 17:10:38.942626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.942618 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" event={"ID":"eb004635-2806-43ea-af5b-28d931fe356f","Type":"ContainerStarted","Data":"fefd81cf12308cafb13391303280551a5dced0c9026858ef261ca45d8db050af"} Jul 28 17:10:38.945055 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.945033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"a259e027b50440f9270844e900ebfa5bcab5b1e54ca1f7bb44588166de5ed796"} Jul 28 17:10:38.945147 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.945099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"4dc135a3927cda578029f0e407f2ec76d6eb657f88b4fb72a43108bb720d2918"} Jul 28 17:10:38.946649 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.946625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f696545-fcmhv" event={"ID":"e59b27f6-a2b3-4dcb-b68c-1efa5325269e","Type":"ContainerStarted","Data":"36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf"} Jul 28 17:10:38.949391 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.949369 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wcwdt" event={"ID":"a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5","Type":"ContainerStarted","Data":"5248bf83b9c7abd8ce70f5ee103473d9e4c25a3496b76116d4a57615870b4e9a"} Jul 28 17:10:38.960479 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.960431 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-s7x2k" podStartSLOduration=4.950091495 podStartE2EDuration="12.960412422s" totalImagesPullingTime="8.010320927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.957824341 +0000 UTC m=+81.023320310" watchObservedRunningTime="2026-07-28 17:10:38.960412422 +0000 UTC m=+81.025908387" Jul 28 17:10:38.997166 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:38.996967 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dfff7fddf-gdf6n" podStartSLOduration=4.911719231 podStartE2EDuration="7.996949675s" totalImagesPullingTime="3.085230444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.993495568 +0000 UTC m=+81.058991533" watchObservedRunningTime="2026-07-28 17:10:38.996949675 +0000 UTC m=+81.062445661" Jul 28 17:10:39.063147 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.063083 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56f696545-fcmhv" podStartSLOduration=7.063064054 podStartE2EDuration="7.063064054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.060068378 +0000 UTC m=+81.125564344" watchObservedRunningTime="2026-07-28 17:10:39.063064054 +0000 UTC m=+81.128560019" Jul 28 17:10:39.943797 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.943213 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wcwdt" podStartSLOduration=9.985742967 podStartE2EDuration="13.943189678s" totalImagesPullingTime="3.880849564s" totalInitContainerRuntime="76.597147ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.123457633 +0000 UTC m=+81.188953598" watchObservedRunningTime="2026-07-28 17:10:39.943189678 +0000 UTC m=+82.008685645" Jul 28 17:10:39.943797 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.943698 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f696545-fcmhv"] Jul 28 17:10:39.963591 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.963024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"0945687913ae12c58c51316d31d9397a08dc2bd8c5d89f4b02e3029f3200cc27"} Jul 28 17:10:39.963591 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.963105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"968de5d1d44c05157dccffc904df4dfdc552620e408a4f3f338c734fd994ee77"} Jul 28 17:10:39.963591 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.963147 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" event={"ID":"ea4fcc8e-4446-47d8-8676-37679d510c18","Type":"ContainerStarted","Data":"eabcd978440f384f15190ca93b7d332fa20e17869495204b00dde86fa76c33ed"} Jul 28 17:10:39.963591 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.963503 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:39.979578 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.979023 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbfdfdb5d-cl2p6"] Jul 28 17:10:39.987130 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:39.987102 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.019574 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.019468 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbfdfdb5d-cl2p6"] Jul 28 17:10:40.091803 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.089526 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" podStartSLOduration=3.7552685649999997 podStartE2EDuration="12.089504063s" totalImagesPullingTime="8.334235498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:40.019611147 +0000 UTC m=+82.085107109" watchObservedRunningTime="2026-07-28 17:10:40.089504063 +0000 UTC m=+82.155000029" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-oauth-config\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-service-ca\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-serving-cert\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-console-config\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119529 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkhd\" (UniqueName: \"kubernetes.io/projected/39aef28f-89c2-461b-896c-929860c31a87-kube-api-access-djkhd\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119597 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-trusted-ca-bundle\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.120401 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.119786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-oauth-serving-cert\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221232 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-console-config\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221399 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkhd\" (UniqueName: \"kubernetes.io/projected/39aef28f-89c2-461b-896c-929860c31a87-kube-api-access-djkhd\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221399 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221293 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-trusted-ca-bundle\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221399 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-oauth-serving-cert\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221568 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-oauth-config\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221568 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-service-ca\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.221568 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.221466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-serving-cert\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.222152 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.222129 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-console-config\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.222250 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.222193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-oauth-serving-cert\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.222837 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.222769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-trusted-ca-bundle\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.223279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.223235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-service-ca\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.226608 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.226584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-oauth-config\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.226715 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.226680 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-serving-cert\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.231685 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.231633 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkhd\" (UniqueName: \"kubernetes.io/projected/39aef28f-89c2-461b-896c-929860c31a87-kube-api-access-djkhd\") pod \"console-cbfdfdb5d-cl2p6\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.304851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.304811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:40.468964 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:40.468926 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbfdfdb5d-cl2p6"] Jul 28 17:10:42.201365 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:10:42.201329 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39aef28f_89c2_461b_896c_929860c31a87.slice/crio-1728268c976248d93a335eeeb9aaba0246cb92d8ed65b557b7126dc27f128da8 WatchSource:0}: Error finding container 1728268c976248d93a335eeeb9aaba0246cb92d8ed65b557b7126dc27f128da8: Status 404 returned error can't find the container with id 1728268c976248d93a335eeeb9aaba0246cb92d8ed65b557b7126dc27f128da8 Jul 28 17:10:42.978656 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:42.978617 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbfdfdb5d-cl2p6" event={"ID":"39aef28f-89c2-461b-896c-929860c31a87","Type":"ContainerStarted","Data":"1728268c976248d93a335eeeb9aaba0246cb92d8ed65b557b7126dc27f128da8"} Jul 28 17:10:43.080225 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:43.080181 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:10:44.990511 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.990467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlnz9" event={"ID":"bf7c933c-1dcc-4849-80b8-c5a349f19008","Type":"ContainerStarted","Data":"7f5c67dd368773767d1a2b661dbbee196deb65a8da9f4112ae2463ca673ed4d5"} Jul 28 17:10:44.993895 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.993845 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"b21085213cc5202bb73baf01c64d1538287b93fe3e7a4a0d5e94008813b8d3b9"} Jul 28 17:10:44.994024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.993941 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"e2447b4ef3d66d026c455fcda7d194e0a6aac13cf757b1825789cf1594ecbd20"} Jul 28 17:10:44.994024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.993989 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"191407144adb275e6b74c8fa2eb204d908f394846d8ddb86ac850c5bb623480c"} Jul 28 17:10:44.994103 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.994029 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"33513c2c468f893339a71cc498c8708f4da49cf877baf2e9a610929e17a26db5"} Jul 28 17:10:44.994103 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.994072 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"33f0520dbeaf608b08e29ffa871c71a76e66b89fc81ebd712998d8ed396e1c18"} Jul 28 17:10:44.994186 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.994110 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"55c73256-263d-47f0-8f4b-dcdd4049e568","Type":"ContainerStarted","Data":"6bc90d729af4f6c764221086d591027aab972ff5ed93a65130647612db5c7136"} Jul 28 17:10:44.995513 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.995494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbfdfdb5d-cl2p6" event={"ID":"39aef28f-89c2-461b-896c-929860c31a87","Type":"ContainerStarted","Data":"a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad"} Jul 28 17:10:44.998644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.998613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"df767da83787032196febf727e04f13c2bd3b811c65568490f64782c792fae9e"} Jul 28 17:10:44.998730 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.998677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"233deb86f5c770be523362c67f6b2a246adf24e8d9131bb751427f3f43b8d20d"} Jul 28 17:10:44.998730 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.998707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"78bc0c35608427b2016ace6c18d7f64fd596bf6556440fa4708cfce8bc724850"} Jul 28 17:10:44.998823 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.998731 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"257b984a36fbcb44b0f568ea4094b13503885663f7bff63cc5d37b4c15ae8391"} Jul 28 17:10:44.998823 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.998759 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"66519ee3c44c43c3ace2f66eea1ea72e5cbdf467b7b409a648fe605def4706f5"} Jul 28 17:10:44.998823 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:44.998793 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a7acbbb3-8e15-47db-8361-31f545deba15","Type":"ContainerStarted","Data":"c434ba5348d8cc4811ad2d821be65e6b02fc873fe268514246a87c2964c430e6"} Jul 28 17:10:45.070520 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:45.070471 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zlnz9" podStartSLOduration=6.07310582 podStartE2EDuration="27.070456327s" totalImagesPullingTime="20.997350507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.026896793 +0000 UTC m=+87.092392754" watchObservedRunningTime="2026-07-28 17:10:45.070456327 +0000 UTC m=+87.135952293" Jul 28 17:10:45.070828 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:45.070807 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.956904814 podStartE2EDuration="13.070797851s" totalImagesPullingTime="5.073058993s" totalInitContainerRuntime="40.834044ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.069043785 +0000 UTC m=+87.134539764" watchObservedRunningTime="2026-07-28 17:10:45.070797851 +0000 UTC m=+87.136293819" Jul 28 17:10:45.094125 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:45.094074 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cbfdfdb5d-cl2p6" podStartSLOduration=6.094058705 podStartE2EDuration="6.094058705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.092704254 +0000 UTC m=+87.158200220" watchObservedRunningTime="2026-07-28 17:10:45.094058705 +0000 UTC m=+87.159554671" Jul 28 17:10:45.127738 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:45.127687 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.473091817 podStartE2EDuration="18.127670395s" totalImagesPullingTime="12.632140331s" totalInitContainerRuntime="22.438247ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.125525295 +0000 UTC m=+87.191021261" watchObservedRunningTime="2026-07-28 17:10:45.127670395 +0000 UTC m=+87.193166359" Jul 28 17:10:45.972680 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:45.972650 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d56d77cf6-wgvb7" Jul 28 17:10:47.802896 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:47.802849 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:10:48.876770 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:48.876704 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86f454b754-6629g" podUID="03030ad1-2da2-49f0-85c0-93ea9ead8259" containerName="console" containerID="cri-o://0b4c8b4227ddd392d22fe251661cd5404cc076eb3e85dfd90a303921d7a30b2f" gracePeriod=15 Jul 28 17:10:49.019630 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.019597 2567 generic.go:356] "Generic (PLEG): container finished" podID="03030ad1-2da2-49f0-85c0-93ea9ead8259" containerID="0b4c8b4227ddd392d22fe251661cd5404cc076eb3e85dfd90a303921d7a30b2f" exitCode=0 Jul 28 17:10:49.019766 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.019652 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f454b754-6629g" event={"ID":"03030ad1-2da2-49f0-85c0-93ea9ead8259","Type":"ContainerDied","Data":"0b4c8b4227ddd392d22fe251661cd5404cc076eb3e85dfd90a303921d7a30b2f"} Jul 28 17:10:49.158126 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.158101 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:49.226677 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.226643 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-oauth-config\") pod \"03030ad1-2da2-49f0-85c0-93ea9ead8259\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " Jul 28 17:10:49.226850 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.226685 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-serving-cert\") pod \"03030ad1-2da2-49f0-85c0-93ea9ead8259\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " Jul 28 17:10:49.226850 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.226711 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4vj4\" (UniqueName: \"kubernetes.io/projected/03030ad1-2da2-49f0-85c0-93ea9ead8259-kube-api-access-f4vj4\") pod \"03030ad1-2da2-49f0-85c0-93ea9ead8259\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " Jul 28 17:10:49.226850 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.226730 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-config\") pod \"03030ad1-2da2-49f0-85c0-93ea9ead8259\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " Jul 28 17:10:49.226850 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.226760 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-oauth-serving-cert\") pod \"03030ad1-2da2-49f0-85c0-93ea9ead8259\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " Jul 28 17:10:49.226850 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.226811 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-service-ca\") pod \"03030ad1-2da2-49f0-85c0-93ea9ead8259\" (UID: \"03030ad1-2da2-49f0-85c0-93ea9ead8259\") " Jul 28 17:10:49.227340 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.227300 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "03030ad1-2da2-49f0-85c0-93ea9ead8259" (UID: "03030ad1-2da2-49f0-85c0-93ea9ead8259"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:49.227445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.227357 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-config" (OuterVolumeSpecName: "console-config") pod "03030ad1-2da2-49f0-85c0-93ea9ead8259" (UID: "03030ad1-2da2-49f0-85c0-93ea9ead8259"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:49.227445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.227379 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-service-ca" (OuterVolumeSpecName: "service-ca") pod "03030ad1-2da2-49f0-85c0-93ea9ead8259" (UID: "03030ad1-2da2-49f0-85c0-93ea9ead8259"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:49.229262 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.229229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "03030ad1-2da2-49f0-85c0-93ea9ead8259" (UID: "03030ad1-2da2-49f0-85c0-93ea9ead8259"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:49.229423 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.229403 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03030ad1-2da2-49f0-85c0-93ea9ead8259-kube-api-access-f4vj4" (OuterVolumeSpecName: "kube-api-access-f4vj4") pod "03030ad1-2da2-49f0-85c0-93ea9ead8259" (UID: "03030ad1-2da2-49f0-85c0-93ea9ead8259"). InnerVolumeSpecName "kube-api-access-f4vj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:10:49.229463 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.229407 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "03030ad1-2da2-49f0-85c0-93ea9ead8259" (UID: "03030ad1-2da2-49f0-85c0-93ea9ead8259"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:49.327605 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.327568 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-oauth-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:10:49.327605 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.327600 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-service-ca\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:10:49.327605 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.327611 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-oauth-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:10:49.327605 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.327619 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:10:49.327889 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.327628 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4vj4\" (UniqueName: \"kubernetes.io/projected/03030ad1-2da2-49f0-85c0-93ea9ead8259-kube-api-access-f4vj4\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:10:49.327889 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:49.327636 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03030ad1-2da2-49f0-85c0-93ea9ead8259-console-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:10:50.025464 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.025434 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f454b754-6629g" Jul 28 17:10:50.025914 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.025469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f454b754-6629g" event={"ID":"03030ad1-2da2-49f0-85c0-93ea9ead8259","Type":"ContainerDied","Data":"161d2afeb32196f53378cce61f8d2451984d79b64bf31eb56effe7cfce4c344a"} Jul 28 17:10:50.025914 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.025523 2567 scope.go:117] "RemoveContainer" containerID="0b4c8b4227ddd392d22fe251661cd5404cc076eb3e85dfd90a303921d7a30b2f" Jul 28 17:10:50.048642 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.048612 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f454b754-6629g"] Jul 28 17:10:50.051932 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.051903 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86f454b754-6629g"] Jul 28 17:10:50.305303 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.305212 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:50.305303 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.305278 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:50.309936 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.309913 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:50.421047 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.421013 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03030ad1-2da2-49f0-85c0-93ea9ead8259" path="/var/lib/kubelet/pods/03030ad1-2da2-49f0-85c0-93ea9ead8259/volumes" Jul 28 17:10:50.862962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.862925 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:50.863148 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:50.862992 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:10:51.035120 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:51.035093 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:10:51.098005 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:10:51.097968 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d99df865-jkjfs"] Jul 28 17:11:04.877701 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:04.877671 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2bfzx" Jul 28 17:11:05.991182 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:05.991142 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56f696545-fcmhv" podUID="e59b27f6-a2b3-4dcb-b68c-1efa5325269e" containerName="console" containerID="cri-o://36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf" gracePeriod=15 Jul 28 17:11:06.770011 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.769985 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:11:06.886790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886707 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-trusted-ca-bundle\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.886790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886752 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-serving-cert\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.886790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886793 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-oauth-config\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.887087 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886837 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7j7s\" (UniqueName: \"kubernetes.io/projected/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-kube-api-access-g7j7s\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.887087 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886888 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-service-ca\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.887087 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886925 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-config\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.887087 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.886994 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-oauth-serving-cert\") pod \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\" (UID: \"e59b27f6-a2b3-4dcb-b68c-1efa5325269e\") " Jul 28 17:11:06.887286 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.887160 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:06.887493 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.887443 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-config" (OuterVolumeSpecName: "console-config") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:06.887493 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.887450 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-service-ca" (OuterVolumeSpecName: "service-ca") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:06.887493 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.887484 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:06.887691 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.887561 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-trusted-ca-bundle\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:06.889238 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.889211 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:11:06.889332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.889278 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:11:06.889332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.889283 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-kube-api-access-g7j7s" (OuterVolumeSpecName: "kube-api-access-g7j7s") pod "e59b27f6-a2b3-4dcb-b68c-1efa5325269e" (UID: "e59b27f6-a2b3-4dcb-b68c-1efa5325269e"). InnerVolumeSpecName "kube-api-access-g7j7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:11:06.988895 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.988818 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7j7s\" (UniqueName: \"kubernetes.io/projected/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-kube-api-access-g7j7s\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:06.988895 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.988887 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-service-ca\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:06.988895 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.988902 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:06.989131 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.988913 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-oauth-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:06.989131 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.988926 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:06.989131 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:06.988937 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e59b27f6-a2b3-4dcb-b68c-1efa5325269e-console-oauth-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:07.102411 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.102376 2567 generic.go:356] "Generic (PLEG): container finished" podID="e59b27f6-a2b3-4dcb-b68c-1efa5325269e" containerID="36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf" exitCode=0 Jul 28 17:11:07.102856 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.102427 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f696545-fcmhv" Jul 28 17:11:07.102856 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.102463 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f696545-fcmhv" event={"ID":"e59b27f6-a2b3-4dcb-b68c-1efa5325269e","Type":"ContainerDied","Data":"36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf"} Jul 28 17:11:07.102856 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.102525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f696545-fcmhv" event={"ID":"e59b27f6-a2b3-4dcb-b68c-1efa5325269e","Type":"ContainerDied","Data":"82927a52e2383e9c43148330fa7de9dc5ff3c7fe103e9b3a9f4e80d285f6627f"} Jul 28 17:11:07.102856 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.102559 2567 scope.go:117] "RemoveContainer" containerID="36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf" Jul 28 17:11:07.114818 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.114798 2567 scope.go:117] "RemoveContainer" containerID="36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf" Jul 28 17:11:07.115110 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:11:07.115091 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf\": container with ID starting with 36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf not found: ID does not exist" containerID="36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf" Jul 28 17:11:07.115162 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.115117 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf"} err="failed to get container status \"36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf\": rpc error: code = NotFound desc = could not find container \"36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf\": container with ID starting with 36d17b1ca9892afcb478f4d12f7a0b20df7f36847213d08f52b7a7e47ca16caf not found: ID does not exist" Jul 28 17:11:07.129834 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.129803 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f696545-fcmhv"] Jul 28 17:11:07.132279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:07.132259 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f696545-fcmhv"] Jul 28 17:11:08.420601 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:08.420564 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e59b27f6-a2b3-4dcb-b68c-1efa5325269e" path="/var/lib/kubelet/pods/e59b27f6-a2b3-4dcb-b68c-1efa5325269e/volumes" Jul 28 17:11:10.868215 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:10.868182 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:11:10.872046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:10.872020 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77b78d6768-nbdds" Jul 28 17:11:16.117229 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.117188 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54d99df865-jkjfs" podUID="6ddc3d4f-180d-47b9-868f-356568a5cf38" containerName="console" containerID="cri-o://b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d" gracePeriod=15 Jul 28 17:11:16.369736 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.369670 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:11:16.473377 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473324 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-oauth-config\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473564 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473431 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5wbs\" (UniqueName: \"kubernetes.io/projected/6ddc3d4f-180d-47b9-868f-356568a5cf38-kube-api-access-d5wbs\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473564 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473465 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-service-ca\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473564 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473500 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-trusted-ca-bundle\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473564 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473531 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-config\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473777 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473612 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-oauth-serving-cert\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473777 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473636 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-serving-cert\") pod \"6ddc3d4f-180d-47b9-868f-356568a5cf38\" (UID: \"6ddc3d4f-180d-47b9-868f-356568a5cf38\") " Jul 28 17:11:16.473962 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473928 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-service-ca" (OuterVolumeSpecName: "service-ca") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:16.474030 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473988 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:16.474030 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473999 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-config" (OuterVolumeSpecName: "console-config") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:16.474030 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.473995 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:16.475767 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.475736 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddc3d4f-180d-47b9-868f-356568a5cf38-kube-api-access-d5wbs" (OuterVolumeSpecName: "kube-api-access-d5wbs") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "kube-api-access-d5wbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:11:16.475767 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.475749 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:11:16.476014 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.475796 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6ddc3d4f-180d-47b9-868f-356568a5cf38" (UID: "6ddc3d4f-180d-47b9-868f-356568a5cf38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:11:16.575121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575078 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:16.575121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575117 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-oauth-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:16.575121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575127 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:16.575121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575136 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ddc3d4f-180d-47b9-868f-356568a5cf38-console-oauth-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:16.575380 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575144 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wbs\" (UniqueName: \"kubernetes.io/projected/6ddc3d4f-180d-47b9-868f-356568a5cf38-kube-api-access-d5wbs\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:16.575380 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575153 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-service-ca\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:16.575380 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:16.575161 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ddc3d4f-180d-47b9-868f-356568a5cf38-trusted-ca-bundle\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:11:17.147997 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.147958 2567 generic.go:356] "Generic (PLEG): container finished" podID="6ddc3d4f-180d-47b9-868f-356568a5cf38" containerID="b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d" exitCode=0 Jul 28 17:11:17.148469 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.148005 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d99df865-jkjfs" Jul 28 17:11:17.148469 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.148043 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d99df865-jkjfs" event={"ID":"6ddc3d4f-180d-47b9-868f-356568a5cf38","Type":"ContainerDied","Data":"b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d"} Jul 28 17:11:17.148469 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.148105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d99df865-jkjfs" event={"ID":"6ddc3d4f-180d-47b9-868f-356568a5cf38","Type":"ContainerDied","Data":"5e964945a187df6156f7bdeeb4a3cbc9d55df99a8daf11666787f86b8fe50fe9"} Jul 28 17:11:17.148469 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.148136 2567 scope.go:117] "RemoveContainer" containerID="b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d" Jul 28 17:11:17.157428 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.157412 2567 scope.go:117] "RemoveContainer" containerID="b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d" Jul 28 17:11:17.157673 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:11:17.157655 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d\": container with ID starting with b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d not found: ID does not exist" containerID="b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d" Jul 28 17:11:17.157721 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.157680 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d"} err="failed to get container status \"b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d\": rpc error: code = NotFound desc = could not find container \"b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d\": container with ID starting with b7081289c3cdcbdf6f7d83a0ee207e0ced460adb07d58d2dc3e36b67d8c06f9d not found: ID does not exist" Jul 28 17:11:17.177694 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.177671 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d99df865-jkjfs"] Jul 28 17:11:17.186706 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:17.186682 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d99df865-jkjfs"] Jul 28 17:11:18.420925 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:18.420889 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddc3d4f-180d-47b9-868f-356568a5cf38" path="/var/lib/kubelet/pods/6ddc3d4f-180d-47b9-868f-356568a5cf38/volumes" Jul 28 17:11:32.802681 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:32.802629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:11:32.822894 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:32.822854 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:11:33.227517 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:11:33.227490 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 17:12:00.239156 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239123 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857bf459fb-zb95r"] Jul 28 17:12:00.239569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239479 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddc3d4f-180d-47b9-868f-356568a5cf38" containerName="console" Jul 28 17:12:00.239569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239490 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddc3d4f-180d-47b9-868f-356568a5cf38" containerName="console" Jul 28 17:12:00.239569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239506 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03030ad1-2da2-49f0-85c0-93ea9ead8259" containerName="console" Jul 28 17:12:00.239569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239513 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="03030ad1-2da2-49f0-85c0-93ea9ead8259" containerName="console" Jul 28 17:12:00.239569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239531 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59b27f6-a2b3-4dcb-b68c-1efa5325269e" containerName="console" Jul 28 17:12:00.239569 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239536 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59b27f6-a2b3-4dcb-b68c-1efa5325269e" containerName="console" Jul 28 17:12:00.239753 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239585 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59b27f6-a2b3-4dcb-b68c-1efa5325269e" containerName="console" Jul 28 17:12:00.239753 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239595 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="03030ad1-2da2-49f0-85c0-93ea9ead8259" containerName="console" Jul 28 17:12:00.239753 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.239603 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddc3d4f-180d-47b9-868f-356568a5cf38" containerName="console" Jul 28 17:12:00.244140 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.244121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.321293 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.321259 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857bf459fb-zb95r"] Jul 28 17:12:00.364792 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.364763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-oauth-config\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.364792 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.364794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-trusted-ca-bundle\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.365003 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.364841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-oauth-serving-cert\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.365003 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.364886 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc52x\" (UniqueName: \"kubernetes.io/projected/af723f5d-e527-4f0a-82ed-60dc769be9c9-kube-api-access-rc52x\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.365003 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.364903 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-service-ca\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.365098 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.364991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-serving-cert\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.365098 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.365031 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-config\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.466445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-oauth-config\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.466445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-trusted-ca-bundle\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.466699 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-oauth-serving-cert\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.466775 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466753 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc52x\" (UniqueName: \"kubernetes.io/projected/af723f5d-e527-4f0a-82ed-60dc769be9c9-kube-api-access-rc52x\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.466834 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466802 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-service-ca\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.466980 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466960 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-serving-cert\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.467036 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.466998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-config\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.467409 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.467356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-oauth-serving-cert\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.467543 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.467425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-trusted-ca-bundle\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.467643 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.467618 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-config\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.467743 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.467645 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-service-ca\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.469315 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.469300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-oauth-config\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.469588 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.469568 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-serving-cert\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.476284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.476261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc52x\" (UniqueName: \"kubernetes.io/projected/af723f5d-e527-4f0a-82ed-60dc769be9c9-kube-api-access-rc52x\") pod \"console-857bf459fb-zb95r\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.552783 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.552703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:00.685952 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:00.685919 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857bf459fb-zb95r"] Jul 28 17:12:00.689267 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:12:00.689233 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf723f5d_e527_4f0a_82ed_60dc769be9c9.slice/crio-7fd3bfeaec57cd2660924b441a8d04717e723e0dcb5f9b2438d47b5164537b21 WatchSource:0}: Error finding container 7fd3bfeaec57cd2660924b441a8d04717e723e0dcb5f9b2438d47b5164537b21: Status 404 returned error can't find the container with id 7fd3bfeaec57cd2660924b441a8d04717e723e0dcb5f9b2438d47b5164537b21 Jul 28 17:12:01.329579 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:01.329536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857bf459fb-zb95r" event={"ID":"af723f5d-e527-4f0a-82ed-60dc769be9c9","Type":"ContainerStarted","Data":"bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073"} Jul 28 17:12:01.329579 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:01.329590 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857bf459fb-zb95r" event={"ID":"af723f5d-e527-4f0a-82ed-60dc769be9c9","Type":"ContainerStarted","Data":"7fd3bfeaec57cd2660924b441a8d04717e723e0dcb5f9b2438d47b5164537b21"} Jul 28 17:12:10.553660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:10.553572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:10.553660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:10.553645 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:10.558290 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:10.558269 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:10.579686 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:10.579635 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857bf459fb-zb95r" podStartSLOduration=10.579620887 podStartE2EDuration="10.579620887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:01.393666469 +0000 UTC m=+163.459162434" watchObservedRunningTime="2026-07-28 17:12:10.579620887 +0000 UTC m=+172.645116852" Jul 28 17:12:11.373517 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:11.373485 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:12:11.452739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:11.452710 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbfdfdb5d-cl2p6"] Jul 28 17:12:36.471809 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.471767 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cbfdfdb5d-cl2p6" podUID="39aef28f-89c2-461b-896c-929860c31a87" containerName="console" containerID="cri-o://a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad" gracePeriod=15 Jul 28 17:12:36.710291 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.710263 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:12:36.790419 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790326 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-oauth-config\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790419 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790374 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-oauth-serving-cert\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790455 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djkhd\" (UniqueName: \"kubernetes.io/projected/39aef28f-89c2-461b-896c-929860c31a87-kube-api-access-djkhd\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-service-ca\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790521 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-console-config\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790660 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790592 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-trusted-ca-bundle\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790896 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790663 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-serving-cert\") pod \"39aef28f-89c2-461b-896c-929860c31a87\" (UID: \"39aef28f-89c2-461b-896c-929860c31a87\") " Jul 28 17:12:36.790965 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790909 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:36.791018 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790961 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-console-config" (OuterVolumeSpecName: "console-config") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:36.791018 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.790968 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-service-ca" (OuterVolumeSpecName: "service-ca") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:36.791166 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.791049 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-service-ca\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:36.791166 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.791050 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:36.791166 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.791066 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-console-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:36.791166 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.791079 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-oauth-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:36.792593 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.792569 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aef28f-89c2-461b-896c-929860c31a87-kube-api-access-djkhd" (OuterVolumeSpecName: "kube-api-access-djkhd") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "kube-api-access-djkhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:12:36.792710 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.792594 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:12:36.792710 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.792702 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "39aef28f-89c2-461b-896c-929860c31a87" (UID: "39aef28f-89c2-461b-896c-929860c31a87"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:12:36.892514 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.892475 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-oauth-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:36.892514 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.892507 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkhd\" (UniqueName: \"kubernetes.io/projected/39aef28f-89c2-461b-896c-929860c31a87-kube-api-access-djkhd\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:36.892514 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.892519 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39aef28f-89c2-461b-896c-929860c31a87-trusted-ca-bundle\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:36.892749 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:36.892529 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39aef28f-89c2-461b-896c-929860c31a87-console-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:12:37.483063 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.483022 2567 generic.go:356] "Generic (PLEG): container finished" podID="39aef28f-89c2-461b-896c-929860c31a87" containerID="a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad" exitCode=0 Jul 28 17:12:37.483456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.483078 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbfdfdb5d-cl2p6" Jul 28 17:12:37.483456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.483094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbfdfdb5d-cl2p6" event={"ID":"39aef28f-89c2-461b-896c-929860c31a87","Type":"ContainerDied","Data":"a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad"} Jul 28 17:12:37.483456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.483170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbfdfdb5d-cl2p6" event={"ID":"39aef28f-89c2-461b-896c-929860c31a87","Type":"ContainerDied","Data":"1728268c976248d93a335eeeb9aaba0246cb92d8ed65b557b7126dc27f128da8"} Jul 28 17:12:37.483456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.483196 2567 scope.go:117] "RemoveContainer" containerID="a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad" Jul 28 17:12:37.493996 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.493978 2567 scope.go:117] "RemoveContainer" containerID="a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad" Jul 28 17:12:37.494260 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:12:37.494240 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad\": container with ID starting with a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad not found: ID does not exist" containerID="a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad" Jul 28 17:12:37.494306 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.494267 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad"} err="failed to get container status \"a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad\": rpc error: code = NotFound desc = could not find container \"a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad\": container with ID starting with a142a7aa6d4e606032e0109aa279cacfd9db9df5cf91f1dff0bde3ce6e3217ad not found: ID does not exist" Jul 28 17:12:37.516832 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.516808 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbfdfdb5d-cl2p6"] Jul 28 17:12:37.518292 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:37.518269 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbfdfdb5d-cl2p6"] Jul 28 17:12:38.420358 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:12:38.420317 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39aef28f-89c2-461b-896c-929860c31a87" path="/var/lib/kubelet/pods/39aef28f-89c2-461b-896c-929860c31a87/volumes" Jul 28 17:14:18.243849 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:14:18.243826 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 17:15:48.050289 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.050254 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6f7fcf76-c2gv4"] Jul 28 17:15:48.050757 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.050656 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39aef28f-89c2-461b-896c-929860c31a87" containerName="console" Jul 28 17:15:48.050757 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.050669 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="39aef28f-89c2-461b-896c-929860c31a87" containerName="console" Jul 28 17:15:48.050757 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.050734 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="39aef28f-89c2-461b-896c-929860c31a87" containerName="console" Jul 28 17:15:48.053652 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.053635 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.064573 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.064548 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6f7fcf76-c2gv4"] Jul 28 17:15:48.178057 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-oauth-config\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.178231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-serving-cert\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.178231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178140 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-trusted-ca-bundle\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.178231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-oauth-serving-cert\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.178231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-service-ca\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.178231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvht9\" (UniqueName: \"kubernetes.io/projected/d124af8e-7cf9-4469-aa07-b2ed548c5620-kube-api-access-bvht9\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.178416 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.178283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-config\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279525 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279494 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-config\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279691 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-oauth-config\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279691 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279582 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-serving-cert\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279691 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-trusted-ca-bundle\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279691 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-oauth-serving-cert\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279839 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-service-ca\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.279919 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.279906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvht9\" (UniqueName: \"kubernetes.io/projected/d124af8e-7cf9-4469-aa07-b2ed548c5620-kube-api-access-bvht9\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.280308 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.280279 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-config\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.280546 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.280507 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-oauth-serving-cert\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.280621 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.280518 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-service-ca\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.280621 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.280555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d124af8e-7cf9-4469-aa07-b2ed548c5620-trusted-ca-bundle\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.282102 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.282075 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-serving-cert\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.282194 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.282147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d124af8e-7cf9-4469-aa07-b2ed548c5620-console-oauth-config\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.288304 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.288286 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvht9\" (UniqueName: \"kubernetes.io/projected/d124af8e-7cf9-4469-aa07-b2ed548c5620-kube-api-access-bvht9\") pod \"console-5b6f7fcf76-c2gv4\" (UID: \"d124af8e-7cf9-4469-aa07-b2ed548c5620\") " pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.363029 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.363000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:48.494958 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:48.494928 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6f7fcf76-c2gv4"] Jul 28 17:15:48.498492 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:15:48.498467 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd124af8e_7cf9_4469_aa07_b2ed548c5620.slice/crio-5e136df16b87777d7e5545227c730c97a5abc8ec04d5e3bfe4fec5c9d083efcc WatchSource:0}: Error finding container 5e136df16b87777d7e5545227c730c97a5abc8ec04d5e3bfe4fec5c9d083efcc: Status 404 returned error can't find the container with id 5e136df16b87777d7e5545227c730c97a5abc8ec04d5e3bfe4fec5c9d083efcc Jul 28 17:15:49.272445 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:49.272410 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6f7fcf76-c2gv4" event={"ID":"d124af8e-7cf9-4469-aa07-b2ed548c5620","Type":"ContainerStarted","Data":"9a6dcaceb0d7c7624ae2d554c48bcf000c05e6b6b3a2bee348a58c4523c95de4"} Jul 28 17:15:49.272842 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:49.272466 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6f7fcf76-c2gv4" event={"ID":"d124af8e-7cf9-4469-aa07-b2ed548c5620","Type":"ContainerStarted","Data":"5e136df16b87777d7e5545227c730c97a5abc8ec04d5e3bfe4fec5c9d083efcc"} Jul 28 17:15:58.363226 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:58.363192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:58.363658 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:58.363245 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:58.368096 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:58.368074 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:58.389002 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:58.388950 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b6f7fcf76-c2gv4" podStartSLOduration=10.388935037 podStartE2EDuration="10.388935037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:49.299842777 +0000 UTC m=+391.365338747" watchObservedRunningTime="2026-07-28 17:15:58.388935037 +0000 UTC m=+400.454431084" Jul 28 17:15:59.317148 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:59.317117 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b6f7fcf76-c2gv4" Jul 28 17:15:59.387729 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:15:59.387694 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857bf459fb-zb95r"] Jul 28 17:16:24.408202 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.408140 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-857bf459fb-zb95r" podUID="af723f5d-e527-4f0a-82ed-60dc769be9c9" containerName="console" containerID="cri-o://bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073" gracePeriod=15 Jul 28 17:16:24.659986 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.659933 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:16:24.709682 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.709644 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-oauth-config\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.709890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.709744 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-trusted-ca-bundle\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.709890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.709783 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc52x\" (UniqueName: \"kubernetes.io/projected/af723f5d-e527-4f0a-82ed-60dc769be9c9-kube-api-access-rc52x\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.709890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.709803 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-config\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.709890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.709883 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-service-ca\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.710116 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.709924 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-oauth-serving-cert\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.710116 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.710001 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-serving-cert\") pod \"af723f5d-e527-4f0a-82ed-60dc769be9c9\" (UID: \"af723f5d-e527-4f0a-82ed-60dc769be9c9\") " Jul 28 17:16:24.710227 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.710177 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:16:24.710423 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.710391 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-trusted-ca-bundle\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:24.710551 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.710478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:16:24.710551 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.710497 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-config" (OuterVolumeSpecName: "console-config") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:16:24.710551 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.710514 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:16:24.712055 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.712030 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af723f5d-e527-4f0a-82ed-60dc769be9c9-kube-api-access-rc52x" (OuterVolumeSpecName: "kube-api-access-rc52x") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "kube-api-access-rc52x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:16:24.712163 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.712073 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:16:24.712279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.712262 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af723f5d-e527-4f0a-82ed-60dc769be9c9" (UID: "af723f5d-e527-4f0a-82ed-60dc769be9c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:16:24.811144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.811095 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:24.811144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.811137 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-oauth-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:24.811144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.811148 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc52x\" (UniqueName: \"kubernetes.io/projected/af723f5d-e527-4f0a-82ed-60dc769be9c9-kube-api-access-rc52x\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:24.811144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.811157 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-console-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:24.811144 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.811165 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-service-ca\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:24.811459 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:24.811173 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af723f5d-e527-4f0a-82ed-60dc769be9c9-oauth-serving-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:16:25.417728 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.417696 2567 generic.go:356] "Generic (PLEG): container finished" podID="af723f5d-e527-4f0a-82ed-60dc769be9c9" containerID="bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073" exitCode=0 Jul 28 17:16:25.418184 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.417737 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857bf459fb-zb95r" event={"ID":"af723f5d-e527-4f0a-82ed-60dc769be9c9","Type":"ContainerDied","Data":"bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073"} Jul 28 17:16:25.418184 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.417745 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857bf459fb-zb95r" Jul 28 17:16:25.418184 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.417767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857bf459fb-zb95r" event={"ID":"af723f5d-e527-4f0a-82ed-60dc769be9c9","Type":"ContainerDied","Data":"7fd3bfeaec57cd2660924b441a8d04717e723e0dcb5f9b2438d47b5164537b21"} Jul 28 17:16:25.418184 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.417789 2567 scope.go:117] "RemoveContainer" containerID="bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073" Jul 28 17:16:25.426562 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.426545 2567 scope.go:117] "RemoveContainer" containerID="bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073" Jul 28 17:16:25.426810 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:16:25.426793 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073\": container with ID starting with bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073 not found: ID does not exist" containerID="bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073" Jul 28 17:16:25.426858 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.426819 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073"} err="failed to get container status \"bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073\": rpc error: code = NotFound desc = could not find container \"bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073\": container with ID starting with bc059bacc9149ff82b175b6c68e2d8be1228152b310a4e8b274fce8bf7534073 not found: ID does not exist" Jul 28 17:16:25.445520 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.445494 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857bf459fb-zb95r"] Jul 28 17:16:25.449104 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:25.449081 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-857bf459fb-zb95r"] Jul 28 17:16:26.421219 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:26.421180 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af723f5d-e527-4f0a-82ed-60dc769be9c9" path="/var/lib/kubelet/pods/af723f5d-e527-4f0a-82ed-60dc769be9c9/volumes" Jul 28 17:16:57.540553 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.540515 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kwc47"] Jul 28 17:16:57.541031 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.540881 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af723f5d-e527-4f0a-82ed-60dc769be9c9" containerName="console" Jul 28 17:16:57.541031 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.540893 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="af723f5d-e527-4f0a-82ed-60dc769be9c9" containerName="console" Jul 28 17:16:57.541031 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.540947 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="af723f5d-e527-4f0a-82ed-60dc769be9c9" containerName="console" Jul 28 17:16:57.544023 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.544006 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kwc47" Jul 28 17:16:57.547034 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.547001 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 17:16:57.547200 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.547172 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 17:16:57.548392 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.548372 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cl2bb\"" Jul 28 17:16:57.548489 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.548395 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 28 17:16:57.554893 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.554857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kwc47"] Jul 28 17:16:57.602169 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.602137 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzg7j\" (UniqueName: \"kubernetes.io/projected/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-kube-api-access-xzg7j\") pod \"s3-init-kwc47\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " pod="kserve/s3-init-kwc47" Jul 28 17:16:57.602279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.602258 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-model-cache\") pod \"s3-init-kwc47\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " pod="kserve/s3-init-kwc47" Jul 28 17:16:57.703057 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.703018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzg7j\" (UniqueName: \"kubernetes.io/projected/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-kube-api-access-xzg7j\") pod \"s3-init-kwc47\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " pod="kserve/s3-init-kwc47" Jul 28 17:16:57.703235 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.703123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-model-cache\") pod \"s3-init-kwc47\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " pod="kserve/s3-init-kwc47" Jul 28 17:16:57.703469 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.703451 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-model-cache\") pod \"s3-init-kwc47\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " pod="kserve/s3-init-kwc47" Jul 28 17:16:57.712650 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.712616 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzg7j\" (UniqueName: \"kubernetes.io/projected/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-kube-api-access-xzg7j\") pod \"s3-init-kwc47\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " pod="kserve/s3-init-kwc47" Jul 28 17:16:57.860045 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.860017 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kwc47" Jul 28 17:16:57.987781 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.987757 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kwc47"] Jul 28 17:16:57.990298 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:16:57.990272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode036a5e5_4f9f_42f4_ae4f_5b83037487fa.slice/crio-1e4eed2e43d5fe596563453d0b10142e8dd7ff0b9092124f2e835fa9642d5259 WatchSource:0}: Error finding container 1e4eed2e43d5fe596563453d0b10142e8dd7ff0b9092124f2e835fa9642d5259: Status 404 returned error can't find the container with id 1e4eed2e43d5fe596563453d0b10142e8dd7ff0b9092124f2e835fa9642d5259 Jul 28 17:16:57.992153 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:57.992135 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:16:58.553670 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:16:58.553634 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kwc47" event={"ID":"e036a5e5-4f9f-42f4-ae4f-5b83037487fa","Type":"ContainerStarted","Data":"1e4eed2e43d5fe596563453d0b10142e8dd7ff0b9092124f2e835fa9642d5259"} Jul 28 17:17:02.577056 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:17:02.577015 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kwc47" event={"ID":"e036a5e5-4f9f-42f4-ae4f-5b83037487fa","Type":"ContainerStarted","Data":"b51a60fc0e52d5d1accaebe15a50f779538faa6df9d63380f374a400a70ef11f"} Jul 28 17:17:15.636657 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:17:15.636620 2567 generic.go:356] "Generic (PLEG): container finished" podID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerID="b51a60fc0e52d5d1accaebe15a50f779538faa6df9d63380f374a400a70ef11f" exitCode=0 Jul 28 17:17:15.637125 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:17:15.636690 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kwc47" event={"ID":"e036a5e5-4f9f-42f4-ae4f-5b83037487fa","Type":"ContainerDied","Data":"b51a60fc0e52d5d1accaebe15a50f779538faa6df9d63380f374a400a70ef11f"} Jul 28 17:17:19.971474 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:17:19.971451 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 28 17:17:20.664463 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:17:20.664420 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kwc47" event={"ID":"e036a5e5-4f9f-42f4-ae4f-5b83037487fa","Type":"ContainerStarted","Data":"e0d5d1476cd6867566b017a8660ce8f62f581d3030bbba029a71dbfc1ac90911"} Jul 28 17:17:20.687851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:17:20.687804 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-kwc47" podStartSLOduration=1.878012563 podStartE2EDuration="23.687790718s" totalImagesPullingTime="8.269324302s" totalInitContainerRuntime="13.540453853s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:20.685966751 +0000 UTC m=+482.751462730" watchObservedRunningTime="2026-07-28 17:17:20.687790718 +0000 UTC m=+482.753286682" Jul 28 17:18:25.924731 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:25.924697 2567 generic.go:356] "Generic (PLEG): container finished" podID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerID="e0d5d1476cd6867566b017a8660ce8f62f581d3030bbba029a71dbfc1ac90911" exitCode=0 Jul 28 17:18:25.925169 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:25.924768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kwc47" event={"ID":"e036a5e5-4f9f-42f4-ae4f-5b83037487fa","Type":"ContainerDied","Data":"e0d5d1476cd6867566b017a8660ce8f62f581d3030bbba029a71dbfc1ac90911"} Jul 28 17:18:27.053678 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.053653 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kwc47" Jul 28 17:18:27.099586 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.099555 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzg7j\" (UniqueName: \"kubernetes.io/projected/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-kube-api-access-xzg7j\") pod \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " Jul 28 17:18:27.099741 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.099622 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-model-cache\") pod \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\" (UID: \"e036a5e5-4f9f-42f4-ae4f-5b83037487fa\") " Jul 28 17:18:27.101745 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.101712 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-kube-api-access-xzg7j" (OuterVolumeSpecName: "kube-api-access-xzg7j") pod "e036a5e5-4f9f-42f4-ae4f-5b83037487fa" (UID: "e036a5e5-4f9f-42f4-ae4f-5b83037487fa"). InnerVolumeSpecName "kube-api-access-xzg7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:18:27.122647 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.122618 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "e036a5e5-4f9f-42f4-ae4f-5b83037487fa" (UID: "e036a5e5-4f9f-42f4-ae4f-5b83037487fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:18:27.200536 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.200460 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-model-cache\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:18:27.200536 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.200488 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzg7j\" (UniqueName: \"kubernetes.io/projected/e036a5e5-4f9f-42f4-ae4f-5b83037487fa-kube-api-access-xzg7j\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:18:27.933933 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.933899 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kwc47" event={"ID":"e036a5e5-4f9f-42f4-ae4f-5b83037487fa","Type":"ContainerDied","Data":"1e4eed2e43d5fe596563453d0b10142e8dd7ff0b9092124f2e835fa9642d5259"} Jul 28 17:18:27.933933 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.933924 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kwc47" Jul 28 17:18:27.933933 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:27.933938 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e4eed2e43d5fe596563453d0b10142e8dd7ff0b9092124f2e835fa9642d5259" Jul 28 17:18:38.443808 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.443769 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-cwl9t"] Jul 28 17:18:38.444207 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.444121 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerName="s3-init" Jul 28 17:18:38.444207 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.444133 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerName="s3-init" Jul 28 17:18:38.444207 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.444146 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerName="download-hf-model" Jul 28 17:18:38.444207 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.444153 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerName="download-hf-model" Jul 28 17:18:38.444336 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.444215 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e036a5e5-4f9f-42f4-ae4f-5b83037487fa" containerName="s3-init" Jul 28 17:18:38.447503 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.447487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.450600 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.450579 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Jul 28 17:18:38.450846 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.450830 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cl2bb\"" Jul 28 17:18:38.451854 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.451837 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 17:18:38.451954 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.451885 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 17:18:38.457313 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.457291 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-cwl9t"] Jul 28 17:18:38.498436 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.498405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05df1e67-595f-476d-9ac8-a93df2e2785e-model-cache\") pod \"s3-tls-init-custom-cwl9t\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.498565 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.498506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwg8\" (UniqueName: \"kubernetes.io/projected/05df1e67-595f-476d-9ac8-a93df2e2785e-kube-api-access-6hwg8\") pod \"s3-tls-init-custom-cwl9t\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.598922 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.598892 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwg8\" (UniqueName: \"kubernetes.io/projected/05df1e67-595f-476d-9ac8-a93df2e2785e-kube-api-access-6hwg8\") pod \"s3-tls-init-custom-cwl9t\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.599051 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.598940 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05df1e67-595f-476d-9ac8-a93df2e2785e-model-cache\") pod \"s3-tls-init-custom-cwl9t\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.599284 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.599268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05df1e67-595f-476d-9ac8-a93df2e2785e-model-cache\") pod \"s3-tls-init-custom-cwl9t\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.608239 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.608213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwg8\" (UniqueName: \"kubernetes.io/projected/05df1e67-595f-476d-9ac8-a93df2e2785e-kube-api-access-6hwg8\") pod \"s3-tls-init-custom-cwl9t\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.763722 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.763649 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:38.898746 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.898567 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-cwl9t"] Jul 28 17:18:38.901694 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:18:38.901665 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05df1e67_595f_476d_9ac8_a93df2e2785e.slice/crio-9214efc2236d247cd6c73cf132c4ed4fc119f495713b67710564c2638961db99 WatchSource:0}: Error finding container 9214efc2236d247cd6c73cf132c4ed4fc119f495713b67710564c2638961db99: Status 404 returned error can't find the container with id 9214efc2236d247cd6c73cf132c4ed4fc119f495713b67710564c2638961db99 Jul 28 17:18:38.979112 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.979079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cwl9t" event={"ID":"05df1e67-595f-476d-9ac8-a93df2e2785e","Type":"ContainerStarted","Data":"a8916aaeabef2ec36725d299b461f399d65af669f669e74c5e0d80f140b52625"} Jul 28 17:18:38.979242 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:38.979129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cwl9t" event={"ID":"05df1e67-595f-476d-9ac8-a93df2e2785e","Type":"ContainerStarted","Data":"9214efc2236d247cd6c73cf132c4ed4fc119f495713b67710564c2638961db99"} Jul 28 17:18:49.031040 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:49.031003 2567 generic.go:356] "Generic (PLEG): container finished" podID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerID="a8916aaeabef2ec36725d299b461f399d65af669f669e74c5e0d80f140b52625" exitCode=0 Jul 28 17:18:49.031434 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:49.031074 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cwl9t" event={"ID":"05df1e67-595f-476d-9ac8-a93df2e2785e","Type":"ContainerDied","Data":"a8916aaeabef2ec36725d299b461f399d65af669f669e74c5e0d80f140b52625"} Jul 28 17:18:50.043204 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:50.043169 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cwl9t" event={"ID":"05df1e67-595f-476d-9ac8-a93df2e2785e","Type":"ContainerStarted","Data":"a14f934cf3a21b079d51934bd51098db4a4c152eb49c4eeb6d842f8063a49e47"} Jul 28 17:18:50.068417 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:50.068362 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-cwl9t" podStartSLOduration=2.485510265 podStartE2EDuration="12.068347161s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.582836896s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:50.067558529 +0000 UTC m=+572.133054497" watchObservedRunningTime="2026-07-28 17:18:50.068347161 +0000 UTC m=+572.133843188" Jul 28 17:18:55.067482 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:55.067390 2567 generic.go:356] "Generic (PLEG): container finished" podID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerID="a14f934cf3a21b079d51934bd51098db4a4c152eb49c4eeb6d842f8063a49e47" exitCode=0 Jul 28 17:18:55.067482 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:55.067466 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cwl9t" event={"ID":"05df1e67-595f-476d-9ac8-a93df2e2785e","Type":"ContainerDied","Data":"a14f934cf3a21b079d51934bd51098db4a4c152eb49c4eeb6d842f8063a49e47"} Jul 28 17:18:56.210739 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.210718 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:56.361695 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.361669 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hwg8\" (UniqueName: \"kubernetes.io/projected/05df1e67-595f-476d-9ac8-a93df2e2785e-kube-api-access-6hwg8\") pod \"05df1e67-595f-476d-9ac8-a93df2e2785e\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " Jul 28 17:18:56.361884 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.361770 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05df1e67-595f-476d-9ac8-a93df2e2785e-model-cache\") pod \"05df1e67-595f-476d-9ac8-a93df2e2785e\" (UID: \"05df1e67-595f-476d-9ac8-a93df2e2785e\") " Jul 28 17:18:56.363741 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.363715 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05df1e67-595f-476d-9ac8-a93df2e2785e-kube-api-access-6hwg8" (OuterVolumeSpecName: "kube-api-access-6hwg8") pod "05df1e67-595f-476d-9ac8-a93df2e2785e" (UID: "05df1e67-595f-476d-9ac8-a93df2e2785e"). InnerVolumeSpecName "kube-api-access-6hwg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:18:56.382044 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.382017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05df1e67-595f-476d-9ac8-a93df2e2785e-model-cache" (OuterVolumeSpecName: "model-cache") pod "05df1e67-595f-476d-9ac8-a93df2e2785e" (UID: "05df1e67-595f-476d-9ac8-a93df2e2785e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:18:56.462670 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.462631 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05df1e67-595f-476d-9ac8-a93df2e2785e-model-cache\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:18:56.462820 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:56.462685 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hwg8\" (UniqueName: \"kubernetes.io/projected/05df1e67-595f-476d-9ac8-a93df2e2785e-kube-api-access-6hwg8\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:18:57.077670 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:57.077641 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-cwl9t" Jul 28 17:18:57.077670 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:57.077659 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-cwl9t" event={"ID":"05df1e67-595f-476d-9ac8-a93df2e2785e","Type":"ContainerDied","Data":"9214efc2236d247cd6c73cf132c4ed4fc119f495713b67710564c2638961db99"} Jul 28 17:18:57.077911 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:57.077699 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9214efc2236d247cd6c73cf132c4ed4fc119f495713b67710564c2638961db99" Jul 28 17:18:59.824669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.824636 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4nz22"] Jul 28 17:18:59.825172 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.824990 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerName="download-hf-model" Jul 28 17:18:59.825172 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.825003 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerName="download-hf-model" Jul 28 17:18:59.825172 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.825015 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerName="s3-tls-init-custom" Jul 28 17:18:59.825172 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.825021 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerName="s3-tls-init-custom" Jul 28 17:18:59.825172 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.825091 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="05df1e67-595f-476d-9ac8-a93df2e2785e" containerName="s3-tls-init-custom" Jul 28 17:18:59.828169 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.828152 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:18:59.831068 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.831048 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 17:18:59.831202 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.831050 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Jul 28 17:18:59.832235 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.832218 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cl2bb\"" Jul 28 17:18:59.832325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.832249 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 17:18:59.837781 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.837752 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4nz22"] Jul 28 17:18:59.993557 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.993519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgs8\" (UniqueName: \"kubernetes.io/projected/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-kube-api-access-lmgs8\") pod \"s3-tls-init-serving-4nz22\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:18:59.993729 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:18:59.993586 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-model-cache\") pod \"s3-tls-init-serving-4nz22\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:00.094949 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:00.094847 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-model-cache\") pod \"s3-tls-init-serving-4nz22\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:00.095096 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:00.094967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgs8\" (UniqueName: \"kubernetes.io/projected/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-kube-api-access-lmgs8\") pod \"s3-tls-init-serving-4nz22\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:00.095292 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:00.095270 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-model-cache\") pod \"s3-tls-init-serving-4nz22\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:00.104980 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:00.104952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgs8\" (UniqueName: \"kubernetes.io/projected/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-kube-api-access-lmgs8\") pod \"s3-tls-init-serving-4nz22\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:00.156407 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:00.156378 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:00.289635 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:00.289584 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4nz22"] Jul 28 17:19:00.291966 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:19:00.291938 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3b58c0_81b7_43c8_90b0_b3a2f535b508.slice/crio-e87af6f1c90055c96e39bf1c24c953aca699167514f4f3a304a45d6c2db213e2 WatchSource:0}: Error finding container e87af6f1c90055c96e39bf1c24c953aca699167514f4f3a304a45d6c2db213e2: Status 404 returned error can't find the container with id e87af6f1c90055c96e39bf1c24c953aca699167514f4f3a304a45d6c2db213e2 Jul 28 17:19:01.094329 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:01.094289 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4nz22" event={"ID":"9c3b58c0-81b7-43c8-90b0-b3a2f535b508","Type":"ContainerStarted","Data":"7695293ca5630492b6271942134cd0afb8049c879dfe708911f08ae1290255f4"} Jul 28 17:19:01.094329 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:01.094332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4nz22" event={"ID":"9c3b58c0-81b7-43c8-90b0-b3a2f535b508","Type":"ContainerStarted","Data":"e87af6f1c90055c96e39bf1c24c953aca699167514f4f3a304a45d6c2db213e2"} Jul 28 17:19:06.117729 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:06.117694 2567 generic.go:356] "Generic (PLEG): container finished" podID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerID="7695293ca5630492b6271942134cd0afb8049c879dfe708911f08ae1290255f4" exitCode=0 Jul 28 17:19:06.118136 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:06.117766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4nz22" event={"ID":"9c3b58c0-81b7-43c8-90b0-b3a2f535b508","Type":"ContainerDied","Data":"7695293ca5630492b6271942134cd0afb8049c879dfe708911f08ae1290255f4"} Jul 28 17:19:07.123568 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:07.123535 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4nz22" event={"ID":"9c3b58c0-81b7-43c8-90b0-b3a2f535b508","Type":"ContainerStarted","Data":"2fc521d36af57b6f9ef14319ff476db3f3e7656fdd796a0d6317962d5230b2fe"} Jul 28 17:19:07.149890 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:07.146841 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4nz22" podStartSLOduration=2.993946085 podStartE2EDuration="8.146812179s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.152866094s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:07.144653329 +0000 UTC m=+589.210149297" watchObservedRunningTime="2026-07-28 17:19:07.146812179 +0000 UTC m=+589.212308136" Jul 28 17:19:13.151744 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:13.151707 2567 generic.go:356] "Generic (PLEG): container finished" podID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerID="2fc521d36af57b6f9ef14319ff476db3f3e7656fdd796a0d6317962d5230b2fe" exitCode=0 Jul 28 17:19:13.152114 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:13.151783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4nz22" event={"ID":"9c3b58c0-81b7-43c8-90b0-b3a2f535b508","Type":"ContainerDied","Data":"2fc521d36af57b6f9ef14319ff476db3f3e7656fdd796a0d6317962d5230b2fe"} Jul 28 17:19:14.305898 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.305877 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:14.415349 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.415256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-model-cache\") pod \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " Jul 28 17:19:14.415349 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.415334 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmgs8\" (UniqueName: \"kubernetes.io/projected/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-kube-api-access-lmgs8\") pod \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\" (UID: \"9c3b58c0-81b7-43c8-90b0-b3a2f535b508\") " Jul 28 17:19:14.417381 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.417350 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-kube-api-access-lmgs8" (OuterVolumeSpecName: "kube-api-access-lmgs8") pod "9c3b58c0-81b7-43c8-90b0-b3a2f535b508" (UID: "9c3b58c0-81b7-43c8-90b0-b3a2f535b508"). InnerVolumeSpecName "kube-api-access-lmgs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:19:14.437353 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.437296 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c3b58c0-81b7-43c8-90b0-b3a2f535b508" (UID: "9c3b58c0-81b7-43c8-90b0-b3a2f535b508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:19:14.516602 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.516574 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmgs8\" (UniqueName: \"kubernetes.io/projected/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-kube-api-access-lmgs8\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:19:14.516602 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:14.516599 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b58c0-81b7-43c8-90b0-b3a2f535b508-model-cache\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:19:15.161765 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:15.161736 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4nz22" Jul 28 17:19:15.161765 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:15.161744 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4nz22" event={"ID":"9c3b58c0-81b7-43c8-90b0-b3a2f535b508","Type":"ContainerDied","Data":"e87af6f1c90055c96e39bf1c24c953aca699167514f4f3a304a45d6c2db213e2"} Jul 28 17:19:15.162037 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:19:15.161778 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e87af6f1c90055c96e39bf1c24c953aca699167514f4f3a304a45d6c2db213e2" Jul 28 17:38:51.578305 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.578271 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k"] Jul 28 17:38:51.579024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.578638 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerName="s3-tls-init-serving" Jul 28 17:38:51.579024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.578649 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerName="s3-tls-init-serving" Jul 28 17:38:51.579024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.578672 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerName="download-hf-model" Jul 28 17:38:51.579024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.578677 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerName="download-hf-model" Jul 28 17:38:51.579024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.578747 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3b58c0-81b7-43c8-90b0-b3a2f535b508" containerName="s3-tls-init-serving" Jul 28 17:38:51.581871 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.581847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.584671 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.584643 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Jul 28 17:38:51.584816 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.584780 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 17:38:51.584816 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.584794 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:38:51.584816 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.584780 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\"" Jul 28 17:38:51.585805 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.585782 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Jul 28 17:38:51.585937 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.585806 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 17:38:51.585937 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.585914 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:38:51.594097 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.594076 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k"] Jul 28 17:38:51.672224 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.672193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c53b4e7-e39d-46ab-8060-ebee391060ab-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.672395 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.672234 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgs8\" (UniqueName: \"kubernetes.io/projected/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-api-access-sbgs8\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.672395 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.672266 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c53b4e7-e39d-46ab-8060-ebee391060ab-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.672395 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.672318 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.672395 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.672346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.773202 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.773167 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.773368 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.773204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.773368 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.773265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c53b4e7-e39d-46ab-8060-ebee391060ab-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.773368 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.773292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgs8\" (UniqueName: \"kubernetes.io/projected/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-api-access-sbgs8\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.773368 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.773323 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c53b4e7-e39d-46ab-8060-ebee391060ab-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.773722 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.773699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c53b4e7-e39d-46ab-8060-ebee391060ab-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.774043 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.774021 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.774098 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.774021 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.775734 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.775712 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c53b4e7-e39d-46ab-8060-ebee391060ab-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.781625 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.781596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgs8\" (UniqueName: \"kubernetes.io/projected/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-api-access-sbgs8\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttz7k\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:51.892432 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:51.892400 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:38:52.025746 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:52.025720 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k"] Jul 28 17:38:52.027920 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:38:52.027891 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c53b4e7_e39d_46ab_8060_ebee391060ab.slice/crio-43f38c02890fdbc2cd2e0a0a872510a68ee9a058333c1e15dedcab16b25e5540 WatchSource:0}: Error finding container 43f38c02890fdbc2cd2e0a0a872510a68ee9a058333c1e15dedcab16b25e5540: Status 404 returned error can't find the container with id 43f38c02890fdbc2cd2e0a0a872510a68ee9a058333c1e15dedcab16b25e5540 Jul 28 17:38:52.086906 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:52.086850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerStarted","Data":"e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196"} Jul 28 17:38:52.087036 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:52.086945 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerStarted","Data":"43f38c02890fdbc2cd2e0a0a872510a68ee9a058333c1e15dedcab16b25e5540"} Jul 28 17:38:56.105364 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:56.105329 2567 generic.go:356] "Generic (PLEG): container finished" podID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerID="e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196" exitCode=0 Jul 28 17:38:56.105742 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:56.105394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerDied","Data":"e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196"} Jul 28 17:38:56.106549 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:38:56.106532 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:39:09.178342 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:09.178302 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerStarted","Data":"d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2"} Jul 28 17:39:12.197766 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:12.197726 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerStarted","Data":"3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1"} Jul 28 17:39:12.198180 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:12.197825 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:39:12.219804 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:12.219755 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podStartSLOduration=2.4548786 podStartE2EDuration="21.219741367s" totalImagesPullingTime="15.265309154s" totalInitContainerRuntime="3.499553613s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:12.21781952 +0000 UTC m=+1794.283315489" watchObservedRunningTime="2026-07-28 17:39:12.219741367 +0000 UTC m=+1794.285237332" Jul 28 17:39:13.202583 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:13.202547 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:39:13.203851 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:13.203825 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:39:14.206514 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:14.206474 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:39:19.210906 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:19.210853 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:39:19.211525 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:19.211501 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:39:29.211894 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:29.211837 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:39:39.212113 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:39.212070 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:39:49.212250 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:49.212205 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:39:59.211555 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:39:59.211515 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:40:09.212450 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:09.212407 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:40:19.211515 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:19.211474 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:40:25.416052 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:25.416018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:40:31.766516 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:31.766483 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k"] Jul 28 17:40:31.767046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:31.766777 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" containerID="cri-o://d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2" gracePeriod=30 Jul 28 17:40:31.767046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:31.766877 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kube-rbac-proxy" containerID="cri-o://3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1" gracePeriod=30 Jul 28 17:40:32.551013 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:32.550975 2567 generic.go:356] "Generic (PLEG): container finished" podID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerID="3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1" exitCode=2 Jul 28 17:40:32.551208 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:32.551052 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerDied","Data":"3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1"} Jul 28 17:40:34.207449 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:34.207401 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Jul 28 17:40:35.416645 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:35.416591 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Jul 28 17:40:36.721547 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.721515 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:40:36.830140 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830048 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c53b4e7-e39d-46ab-8060-ebee391060ab-kserve-provision-location\") pod \"6c53b4e7-e39d-46ab-8060-ebee391060ab\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " Jul 28 17:40:36.830140 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830099 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbgs8\" (UniqueName: \"kubernetes.io/projected/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-api-access-sbgs8\") pod \"6c53b4e7-e39d-46ab-8060-ebee391060ab\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " Jul 28 17:40:36.830140 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830137 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c53b4e7-e39d-46ab-8060-ebee391060ab-proxy-tls\") pod \"6c53b4e7-e39d-46ab-8060-ebee391060ab\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " Jul 28 17:40:36.830436 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830167 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-cabundle-cert\") pod \"6c53b4e7-e39d-46ab-8060-ebee391060ab\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " Jul 28 17:40:36.830436 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830322 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-rbac-proxy-sar-config\") pod \"6c53b4e7-e39d-46ab-8060-ebee391060ab\" (UID: \"6c53b4e7-e39d-46ab-8060-ebee391060ab\") " Jul 28 17:40:36.830540 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830504 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c53b4e7-e39d-46ab-8060-ebee391060ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c53b4e7-e39d-46ab-8060-ebee391060ab" (UID: "6c53b4e7-e39d-46ab-8060-ebee391060ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:40:36.830614 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830584 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c53b4e7-e39d-46ab-8060-ebee391060ab" (UID: "6c53b4e7-e39d-46ab-8060-ebee391060ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:40:36.830669 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830590 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c53b4e7-e39d-46ab-8060-ebee391060ab" (UID: "6c53b4e7-e39d-46ab-8060-ebee391060ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:40:36.830709 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.830680 2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c53b4e7-e39d-46ab-8060-ebee391060ab-kserve-provision-location\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:40:36.832372 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.832351 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c53b4e7-e39d-46ab-8060-ebee391060ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c53b4e7-e39d-46ab-8060-ebee391060ab" (UID: "6c53b4e7-e39d-46ab-8060-ebee391060ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:40:36.832480 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.832444 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-api-access-sbgs8" (OuterVolumeSpecName: "kube-api-access-sbgs8") pod "6c53b4e7-e39d-46ab-8060-ebee391060ab" (UID: "6c53b4e7-e39d-46ab-8060-ebee391060ab"). InnerVolumeSpecName "kube-api-access-sbgs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:40:36.931288 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.931254 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbgs8\" (UniqueName: \"kubernetes.io/projected/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-api-access-sbgs8\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:40:36.931288 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.931283 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c53b4e7-e39d-46ab-8060-ebee391060ab-proxy-tls\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:40:36.931288 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.931293 2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-cabundle-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:40:36.931518 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:36.931301 2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c53b4e7-e39d-46ab-8060-ebee391060ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:40:37.575305 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.575269 2567 generic.go:356] "Generic (PLEG): container finished" podID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerID="d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2" exitCode=0 Jul 28 17:40:37.575456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.575319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerDied","Data":"d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2"} Jul 28 17:40:37.575456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.575328 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" Jul 28 17:40:37.575456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.575375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k" event={"ID":"6c53b4e7-e39d-46ab-8060-ebee391060ab","Type":"ContainerDied","Data":"43f38c02890fdbc2cd2e0a0a872510a68ee9a058333c1e15dedcab16b25e5540"} Jul 28 17:40:37.575456 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.575403 2567 scope.go:117] "RemoveContainer" containerID="3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1" Jul 28 17:40:37.584490 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.584468 2567 scope.go:117] "RemoveContainer" containerID="d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2" Jul 28 17:40:37.592121 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.592105 2567 scope.go:117] "RemoveContainer" containerID="e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196" Jul 28 17:40:37.599315 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.599292 2567 scope.go:117] "RemoveContainer" containerID="3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1" Jul 28 17:40:37.599549 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:40:37.599533 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1\": container with ID starting with 3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1 not found: ID does not exist" containerID="3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1" Jul 28 17:40:37.599616 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.599562 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1"} err="failed to get container status \"3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1\": rpc error: code = NotFound desc = could not find container \"3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1\": container with ID starting with 3a1ce2eef57325c3e27fe3cf0f9e07f195edd23d1b93e93fca9b8d3b0a0504c1 not found: ID does not exist" Jul 28 17:40:37.599616 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.599586 2567 scope.go:117] "RemoveContainer" containerID="d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2" Jul 28 17:40:37.599832 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:40:37.599808 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2\": container with ID starting with d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2 not found: ID does not exist" containerID="d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2" Jul 28 17:40:37.599970 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.599842 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2"} err="failed to get container status \"d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2\": rpc error: code = NotFound desc = could not find container \"d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2\": container with ID starting with d2013451663cc47f5fcb42a5355ffa3a2e370e2f5e0b23162e61c44cee4f2ef2 not found: ID does not exist" Jul 28 17:40:37.599970 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.599880 2567 scope.go:117] "RemoveContainer" containerID="e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196" Jul 28 17:40:37.600105 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:40:37.600089 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196\": container with ID starting with e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196 not found: ID does not exist" containerID="e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196" Jul 28 17:40:37.600143 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.600109 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196"} err="failed to get container status \"e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196\": rpc error: code = NotFound desc = could not find container \"e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196\": container with ID starting with e371be48aff048e20a3e26a4b0858c1188b03c524b94f66bb6632da46c01b196 not found: ID does not exist" Jul 28 17:40:37.602709 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.602688 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k"] Jul 28 17:40:37.607640 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:37.607620 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttz7k"] Jul 28 17:40:38.420700 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:40:38.420667 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" path="/var/lib/kubelet/pods/6c53b4e7-e39d-46ab-8060-ebee391060ab/volumes" Jul 28 17:42:07.086543 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086507 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2"] Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086888 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086900 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086919 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kube-rbac-proxy" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086924 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kube-rbac-proxy" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086933 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="storage-initializer" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086939 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="storage-initializer" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086987 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kserve-container" Jul 28 17:42:07.087041 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.086998 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c53b4e7-e39d-46ab-8060-ebee391060ab" containerName="kube-rbac-proxy" Jul 28 17:42:07.090132 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.090115 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.093254 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.093230 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 17:42:07.093254 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.093244 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\"" Jul 28 17:42:07.093432 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.093385 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Jul 28 17:42:07.094419 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.094393 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:42:07.094552 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.094488 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Jul 28 17:42:07.094552 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.094529 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 17:42:07.094671 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.094583 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:42:07.102741 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.102719 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2"] Jul 28 17:42:07.233096 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.233051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.233293 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.233144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-api-access-stshj\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.233293 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.233167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.233293 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.233202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.233293 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.233265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334035 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.333976 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-api-access-stshj\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334035 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334306 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334306 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334306 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334640 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334813 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.334856 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.334834 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.336481 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.336463 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.345038 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.344990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-api-access-stshj\") pod \"isvc-canary-rollback-predictor-b4645bb5c-hvrg2\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.400119 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.400088 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:07.530571 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.530540 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2"] Jul 28 17:42:07.963332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.963294 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerStarted","Data":"219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41"} Jul 28 17:42:07.963516 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:07.963356 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerStarted","Data":"19eba990c4fb25674a53b146b9fabff0f209a9edbef4bead6e553beca61a63ff"} Jul 28 17:42:11.980727 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:11.980693 2567 generic.go:356] "Generic (PLEG): container finished" podID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerID="219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41" exitCode=0 Jul 28 17:42:11.981158 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:11.980774 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerDied","Data":"219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41"} Jul 28 17:42:12.987556 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:12.987516 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerStarted","Data":"8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f"} Jul 28 17:42:12.988164 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:12.987589 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerStarted","Data":"6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa"} Jul 28 17:42:12.988164 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:12.987815 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:13.009621 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:13.009571 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podStartSLOduration=2.295065906 podStartE2EDuration="6.00955748s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714491574s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:13.008267288 +0000 UTC m=+1975.073763252" watchObservedRunningTime="2026-07-28 17:42:13.00955748 +0000 UTC m=+1975.075053445" Jul 28 17:42:13.992336 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:13.992301 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:13.993668 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:13.993640 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:42:14.996644 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:14.996603 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:42:20.001595 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:20.001561 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:42:20.002311 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:20.002277 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:42:30.002517 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:30.002473 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:42:40.003139 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:40.003100 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:42:50.003007 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:42:50.002966 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:43:00.003057 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:00.003015 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:43:10.003052 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:10.003012 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:43:20.003011 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:20.002968 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:43:30.003024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:30.002990 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:43:37.319598 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:37.319566 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2"] Jul 28 17:43:37.320091 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:37.319937 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kube-rbac-proxy" containerID="cri-o://8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f" gracePeriod=30 Jul 28 17:43:37.320091 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:37.319939 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" containerID="cri-o://6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa" gracePeriod=30 Jul 28 17:43:38.360046 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:38.360011 2567 generic.go:356] "Generic (PLEG): container finished" podID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerID="8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f" exitCode=2 Jul 28 17:43:38.360436 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:38.360083 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerDied","Data":"8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f"} Jul 28 17:43:39.997767 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:39.997718 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused" Jul 28 17:43:40.002764 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:40.002738 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Jul 28 17:43:42.156007 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.155983 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:43:42.210916 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.210883 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kserve-provision-location\") pod \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " Jul 28 17:43:42.211070 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.210930 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-rbac-proxy-sar-config\") pod \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " Jul 28 17:43:42.211070 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.210954 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-cabundle-cert\") pod \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " Jul 28 17:43:42.211070 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.210991 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-proxy-tls\") pod \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " Jul 28 17:43:42.211230 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211108 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-api-access-stshj\") pod \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\" (UID: \"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb\") " Jul 28 17:43:42.211282 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211233 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" (UID: "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:43:42.211325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211293 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" (UID: "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:43:42.211325 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211312 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" (UID: "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:43:42.211435 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211414 2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kserve-provision-location\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:43:42.211435 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211432 2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:43:42.211574 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.211443 2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-cabundle-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:43:42.213079 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.213060 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-api-access-stshj" (OuterVolumeSpecName: "kube-api-access-stshj") pod "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" (UID: "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb"). InnerVolumeSpecName "kube-api-access-stshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:43:42.213157 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.213137 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" (UID: "8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:43:42.312190 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.312103 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-proxy-tls\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:43:42.312190 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.312135 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb-kube-api-access-stshj\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:43:42.380118 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.380085 2567 generic.go:356] "Generic (PLEG): container finished" podID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerID="6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa" exitCode=0 Jul 28 17:43:42.380279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.380139 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" Jul 28 17:43:42.380279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.380168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerDied","Data":"6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa"} Jul 28 17:43:42.380279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.380236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2" event={"ID":"8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb","Type":"ContainerDied","Data":"19eba990c4fb25674a53b146b9fabff0f209a9edbef4bead6e553beca61a63ff"} Jul 28 17:43:42.380279 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.380266 2567 scope.go:117] "RemoveContainer" containerID="8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f" Jul 28 17:43:42.389040 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.389026 2567 scope.go:117] "RemoveContainer" containerID="6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa" Jul 28 17:43:42.396320 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.396303 2567 scope.go:117] "RemoveContainer" containerID="219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41" Jul 28 17:43:42.403975 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.403854 2567 scope.go:117] "RemoveContainer" containerID="8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f" Jul 28 17:43:42.404162 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:43:42.404143 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f\": container with ID starting with 8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f not found: ID does not exist" containerID="8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f" Jul 28 17:43:42.404232 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.404175 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f"} err="failed to get container status \"8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f\": rpc error: code = NotFound desc = could not find container \"8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f\": container with ID starting with 8cd7dbf90863b4bf54493d5015882df1ca2d2ed4a56d0ac86f86f141b8b9e63f not found: ID does not exist" Jul 28 17:43:42.404232 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.404197 2567 scope.go:117] "RemoveContainer" containerID="6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa" Jul 28 17:43:42.404491 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:43:42.404472 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa\": container with ID starting with 6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa not found: ID does not exist" containerID="6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa" Jul 28 17:43:42.404538 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.404497 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa"} err="failed to get container status \"6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa\": rpc error: code = NotFound desc = could not find container \"6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa\": container with ID starting with 6f34312ee2d60979e27b4a4d76d4d19ef9ce8a3c3dc2d5386fa2e6816f2dc7aa not found: ID does not exist" Jul 28 17:43:42.404538 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.404512 2567 scope.go:117] "RemoveContainer" containerID="219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41" Jul 28 17:43:42.404744 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:43:42.404730 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41\": container with ID starting with 219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41 not found: ID does not exist" containerID="219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41" Jul 28 17:43:42.404790 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.404749 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41"} err="failed to get container status \"219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41\": rpc error: code = NotFound desc = could not find container \"219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41\": container with ID starting with 219da6edf6b5bc6c30c11036f9c76488d72f262d43214764bfbb250188efde41 not found: ID does not exist" Jul 28 17:43:42.405354 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.405335 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2"] Jul 28 17:43:42.413650 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.413629 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-hvrg2"] Jul 28 17:43:42.420792 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:43:42.420773 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" path="/var/lib/kubelet/pods/8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb/volumes" Jul 28 17:54:34.633884 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.633781 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw"] Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634140 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634152 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634179 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="storage-initializer" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634185 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="storage-initializer" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634191 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kube-rbac-proxy" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634197 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kube-rbac-proxy" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634251 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kserve-container" Jul 28 17:54:34.634332 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.634261 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2eb3ee-1c30-4bdb-b1ac-019b20878bdb" containerName="kube-rbac-proxy" Jul 28 17:54:34.637499 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.637481 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.640335 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.640315 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\"" Jul 28 17:54:34.640488 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.640472 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\"" Jul 28 17:54:34.641062 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.641042 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\"" Jul 28 17:54:34.641389 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.641373 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 17:54:34.641504 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.641484 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 17:54:34.641541 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.641488 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 17:54:34.641771 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.641756 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 17:54:34.657227 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.657202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw"] Jul 28 17:54:34.756224 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.756183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8960738b-6b23-4cbb-a5f6-3cb94197bf65-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.756388 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.756300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99gj\" (UniqueName: \"kubernetes.io/projected/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-api-access-b99gj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.756388 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.756331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.756388 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.756349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.756495 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.756406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857004 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.856972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8960738b-6b23-4cbb-a5f6-3cb94197bf65-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857193 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.857039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99gj\" (UniqueName: \"kubernetes.io/projected/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-api-access-b99gj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857193 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.857063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857193 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.857084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857193 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.857112 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857592 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.857572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.857937 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.857911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.858024 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.858010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.859542 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.859521 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8960738b-6b23-4cbb-a5f6-3cb94197bf65-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.867085 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.867061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99gj\" (UniqueName: \"kubernetes.io/projected/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-api-access-b99gj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-j75tw\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:34.947923 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:34.947844 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:35.103567 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:35.103542 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw"] Jul 28 17:54:35.106184 ip-10-0-134-110 kubenswrapper[2567]: W0728 17:54:35.106156 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8960738b_6b23_4cbb_a5f6_3cb94197bf65.slice/crio-6cf6c3ca3f531842a38b8c0e23e8ba60f71c5f2acf90b4424bd8173f4badba2e WatchSource:0}: Error finding container 6cf6c3ca3f531842a38b8c0e23e8ba60f71c5f2acf90b4424bd8173f4badba2e: Status 404 returned error can't find the container with id 6cf6c3ca3f531842a38b8c0e23e8ba60f71c5f2acf90b4424bd8173f4badba2e Jul 28 17:54:36.104613 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:36.104574 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerStarted","Data":"91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a"} Jul 28 17:54:36.105023 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:36.104637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerStarted","Data":"6cf6c3ca3f531842a38b8c0e23e8ba60f71c5f2acf90b4424bd8173f4badba2e"} Jul 28 17:54:40.123799 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:40.123765 2567 generic.go:356] "Generic (PLEG): container finished" podID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerID="91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a" exitCode=0 Jul 28 17:54:40.123799 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:40.123801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerDied","Data":"91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a"} Jul 28 17:54:40.125231 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:40.125216 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:54:53.188619 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:53.188580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerStarted","Data":"6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426"} Jul 28 17:54:53.189080 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:53.188650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerStarted","Data":"a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229"} Jul 28 17:54:53.189080 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:53.188882 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:53.212493 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:53.212443 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podStartSLOduration=2.155182579 podStartE2EDuration="19.212429003s" totalImagesPullingTime="12.454620081s" totalInitContainerRuntime="4.602626343s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:53.210148476 +0000 UTC m=+2735.275644432" watchObservedRunningTime="2026-07-28 17:54:53.212429003 +0000 UTC m=+2735.277924967" Jul 28 17:54:54.192510 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:54.192482 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:54:54.193452 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:54.193425 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 28 17:54:55.196764 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:54:55.196714 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 28 17:55:00.203078 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:00.203044 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:55:00.203763 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:00.203726 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 28 17:55:10.204420 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:10.204376 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 28 17:55:20.204058 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:20.204018 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 28 17:55:30.204545 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:30.204508 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Jul 28 17:55:40.204838 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:40.204808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:55:46.376037 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:46.375998 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw"] Jul 28 17:55:46.376542 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:46.376302 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kube-rbac-proxy" containerID="cri-o://6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426" gracePeriod=30 Jul 28 17:55:46.376542 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:46.376302 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" containerID="cri-o://a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229" gracePeriod=30 Jul 28 17:55:47.434069 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:47.434035 2567 generic.go:356] "Generic (PLEG): container finished" podID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerID="6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426" exitCode=2 Jul 28 17:55:47.434467 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:47.434113 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerDied","Data":"6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426"} Jul 28 17:55:49.226803 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.226775 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:55:49.305438 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305399 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8960738b-6b23-4cbb-a5f6-3cb94197bf65-proxy-tls\") pod \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " Jul 28 17:55:49.305626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305464 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-cabundle-cert\") pod \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " Jul 28 17:55:49.305626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305499 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-rbac-proxy-sar-config\") pod \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " Jul 28 17:55:49.305626 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305556 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b99gj\" (UniqueName: \"kubernetes.io/projected/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-api-access-b99gj\") pod \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " Jul 28 17:55:49.305806 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305633 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kserve-provision-location\") pod \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\" (UID: \"8960738b-6b23-4cbb-a5f6-3cb94197bf65\") " Jul 28 17:55:49.305961 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305929 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8960738b-6b23-4cbb-a5f6-3cb94197bf65" (UID: "8960738b-6b23-4cbb-a5f6-3cb94197bf65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:55:49.305961 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.305939 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8960738b-6b23-4cbb-a5f6-3cb94197bf65" (UID: "8960738b-6b23-4cbb-a5f6-3cb94197bf65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:55:49.306173 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.306011 2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:55:49.306173 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.306027 2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8960738b-6b23-4cbb-a5f6-3cb94197bf65-cabundle-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:55:49.307603 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.307583 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-api-access-b99gj" (OuterVolumeSpecName: "kube-api-access-b99gj") pod "8960738b-6b23-4cbb-a5f6-3cb94197bf65" (UID: "8960738b-6b23-4cbb-a5f6-3cb94197bf65"). InnerVolumeSpecName "kube-api-access-b99gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:55:49.307700 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.307606 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8960738b-6b23-4cbb-a5f6-3cb94197bf65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8960738b-6b23-4cbb-a5f6-3cb94197bf65" (UID: "8960738b-6b23-4cbb-a5f6-3cb94197bf65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:55:49.315092 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.315067 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8960738b-6b23-4cbb-a5f6-3cb94197bf65" (UID: "8960738b-6b23-4cbb-a5f6-3cb94197bf65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:55:49.407356 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.407321 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99gj\" (UniqueName: \"kubernetes.io/projected/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kube-api-access-b99gj\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:55:49.407356 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.407350 2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8960738b-6b23-4cbb-a5f6-3cb94197bf65-kserve-provision-location\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:55:49.407356 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.407361 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8960738b-6b23-4cbb-a5f6-3cb94197bf65-proxy-tls\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 17:55:49.444300 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.444271 2567 generic.go:356] "Generic (PLEG): container finished" podID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerID="a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229" exitCode=0 Jul 28 17:55:49.444438 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.444329 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" Jul 28 17:55:49.444438 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.444361 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerDied","Data":"a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229"} Jul 28 17:55:49.444438 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.444418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw" event={"ID":"8960738b-6b23-4cbb-a5f6-3cb94197bf65","Type":"ContainerDied","Data":"6cf6c3ca3f531842a38b8c0e23e8ba60f71c5f2acf90b4424bd8173f4badba2e"} Jul 28 17:55:49.444543 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.444448 2567 scope.go:117] "RemoveContainer" containerID="6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426" Jul 28 17:55:49.456932 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.456857 2567 scope.go:117] "RemoveContainer" containerID="a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229" Jul 28 17:55:49.466637 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.466617 2567 scope.go:117] "RemoveContainer" containerID="91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a" Jul 28 17:55:49.474166 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.474146 2567 scope.go:117] "RemoveContainer" containerID="6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426" Jul 28 17:55:49.474430 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:55:49.474410 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426\": container with ID starting with 6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426 not found: ID does not exist" containerID="6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426" Jul 28 17:55:49.474473 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.474439 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426"} err="failed to get container status \"6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426\": rpc error: code = NotFound desc = could not find container \"6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426\": container with ID starting with 6469cb47475f405c9dd28ed8dba94cf94588a40e833210967d3ef9c0a3503426 not found: ID does not exist" Jul 28 17:55:49.474473 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.474457 2567 scope.go:117] "RemoveContainer" containerID="a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229" Jul 28 17:55:49.474702 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:55:49.474684 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229\": container with ID starting with a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229 not found: ID does not exist" containerID="a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229" Jul 28 17:55:49.474783 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.474708 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229"} err="failed to get container status \"a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229\": rpc error: code = NotFound desc = could not find container \"a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229\": container with ID starting with a9c4026ab01036539c14943755f67efc502e0b035a12f270b2892ddc4fed5229 not found: ID does not exist" Jul 28 17:55:49.474783 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.474722 2567 scope.go:117] "RemoveContainer" containerID="91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a" Jul 28 17:55:49.474972 ip-10-0-134-110 kubenswrapper[2567]: E0728 17:55:49.474949 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a\": container with ID starting with 91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a not found: ID does not exist" containerID="91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a" Jul 28 17:55:49.475065 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.474976 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a"} err="failed to get container status \"91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a\": rpc error: code = NotFound desc = could not find container \"91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a\": container with ID starting with 91c5723ee5cb7dbeba86393ba4a445737588b20a00bb4ae8fb7ecc04bf2c7d6a not found: ID does not exist" Jul 28 17:55:49.478688 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.478667 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw"] Jul 28 17:55:49.482690 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:49.482673 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-j75tw"] Jul 28 17:55:50.420485 ip-10-0-134-110 kubenswrapper[2567]: I0728 17:55:50.420452 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" path="/var/lib/kubelet/pods/8960738b-6b23-4cbb-a5f6-3cb94197bf65/volumes" Jul 28 18:18:06.429884 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.429833 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss"] Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430179 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="storage-initializer" Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430192 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="storage-initializer" Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430202 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kube-rbac-proxy" Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430209 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kube-rbac-proxy" Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430234 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430239 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" Jul 28 18:18:06.430305 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430307 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kube-rbac-proxy" Jul 28 18:18:06.430521 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.430316 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8960738b-6b23-4cbb-a5f6-3cb94197bf65" containerName="kserve-container" Jul 28 18:18:06.433565 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.433547 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.436368 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.436336 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 18:18:06.436527 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.436512 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 18:18:06.436587 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.436513 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 18:18:06.436587 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.436513 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\"" Jul 28 18:18:06.437713 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.437694 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\"" Jul 28 18:18:06.437713 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.437709 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 18:18:06.437857 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.437703 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\"" Jul 28 18:18:06.444372 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.444349 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss"] Jul 28 18:18:06.581285 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.581238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6215f0e0-d889-49ee-ab05-911e5eaebd74-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.581461 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.581302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.581461 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.581349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6215f0e0-d889-49ee-ab05-911e5eaebd74-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.581461 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.581378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.581461 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.581417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74bd\" (UniqueName: \"kubernetes.io/projected/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-api-access-s74bd\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682245 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682245 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74bd\" (UniqueName: \"kubernetes.io/projected/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-api-access-s74bd\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682479 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6215f0e0-d889-49ee-ab05-911e5eaebd74-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682479 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682479 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6215f0e0-d889-49ee-ab05-911e5eaebd74-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682781 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6215f0e0-d889-49ee-ab05-911e5eaebd74-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682921 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.682977 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.682959 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.684696 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.684677 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6215f0e0-d889-49ee-ab05-911e5eaebd74-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.692002 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.691979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74bd\" (UniqueName: \"kubernetes.io/projected/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-api-access-s74bd\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.745028 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.744985 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:06.881993 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:06.881964 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss"] Jul 28 18:18:06.884354 ip-10-0-134-110 kubenswrapper[2567]: W0728 18:18:06.884318 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6215f0e0_d889_49ee_ab05_911e5eaebd74.slice/crio-7177847c30f27d6aaa69861420084aca647e823afcf4cbe93bb8915fe0f3721b WatchSource:0}: Error finding container 7177847c30f27d6aaa69861420084aca647e823afcf4cbe93bb8915fe0f3721b: Status 404 returned error can't find the container with id 7177847c30f27d6aaa69861420084aca647e823afcf4cbe93bb8915fe0f3721b Jul 28 18:18:07.108939 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:07.108901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerStarted","Data":"9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae"} Jul 28 18:18:07.109113 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:07.108967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerStarted","Data":"7177847c30f27d6aaa69861420084aca647e823afcf4cbe93bb8915fe0f3721b"} Jul 28 18:18:12.131111 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:12.131077 2567 generic.go:356] "Generic (PLEG): container finished" podID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerID="9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae" exitCode=0 Jul 28 18:18:12.131535 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:12.131134 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerDied","Data":"9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae"} Jul 28 18:18:13.138316 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:13.138281 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerStarted","Data":"f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6"} Jul 28 18:18:13.138819 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:13.138393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerStarted","Data":"0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c"} Jul 28 18:18:13.138819 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:13.138629 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:13.159183 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:13.159138 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" podStartSLOduration=2.139953245 podStartE2EDuration="7.159122817s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.019169572s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:18:13.158503076 +0000 UTC m=+4135.223999041" watchObservedRunningTime="2026-07-28 18:18:13.159122817 +0000 UTC m=+4135.224618783" Jul 28 18:18:14.142993 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:14.142963 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:14.144347 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:14.144318 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 28 18:18:15.148788 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:15.148749 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 28 18:18:20.153638 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:20.153610 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:20.154136 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:20.154110 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Jul 28 18:18:30.154409 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:30.154376 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:38.174832 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:38.174797 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss_6215f0e0-d889-49ee-ab05-911e5eaebd74/kserve-container/0.log" Jul 28 18:18:38.331140 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:38.331107 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss"] Jul 28 18:18:38.331427 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:38.331393 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kube-rbac-proxy" containerID="cri-o://f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6" gracePeriod=30 Jul 28 18:18:38.331579 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:38.331393 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" containerID="cri-o://0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c" gracePeriod=30 Jul 28 18:18:39.263401 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.263362 2567 generic.go:356] "Generic (PLEG): container finished" podID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerID="f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6" exitCode=2 Jul 28 18:18:39.263789 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.263419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerDied","Data":"f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6"} Jul 28 18:18:39.677846 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.677820 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:39.796738 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.796702 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s74bd\" (UniqueName: \"kubernetes.io/projected/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-api-access-s74bd\") pod \"6215f0e0-d889-49ee-ab05-911e5eaebd74\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " Jul 28 18:18:39.796967 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.796766 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6215f0e0-d889-49ee-ab05-911e5eaebd74-proxy-tls\") pod \"6215f0e0-d889-49ee-ab05-911e5eaebd74\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " Jul 28 18:18:39.796967 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.796802 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6215f0e0-d889-49ee-ab05-911e5eaebd74-kserve-provision-location\") pod \"6215f0e0-d889-49ee-ab05-911e5eaebd74\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " Jul 28 18:18:39.796967 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.796822 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-rbac-proxy-sar-config\") pod \"6215f0e0-d889-49ee-ab05-911e5eaebd74\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " Jul 28 18:18:39.796967 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.796894 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-cabundle-cert\") pod \"6215f0e0-d889-49ee-ab05-911e5eaebd74\" (UID: \"6215f0e0-d889-49ee-ab05-911e5eaebd74\") " Jul 28 18:18:39.797296 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.797247 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6215f0e0-d889-49ee-ab05-911e5eaebd74" (UID: "6215f0e0-d889-49ee-ab05-911e5eaebd74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:18:39.797348 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.797289 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6215f0e0-d889-49ee-ab05-911e5eaebd74" (UID: "6215f0e0-d889-49ee-ab05-911e5eaebd74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:18:39.799051 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.799028 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6215f0e0-d889-49ee-ab05-911e5eaebd74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6215f0e0-d889-49ee-ab05-911e5eaebd74" (UID: "6215f0e0-d889-49ee-ab05-911e5eaebd74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:18:39.799123 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.799043 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-api-access-s74bd" (OuterVolumeSpecName: "kube-api-access-s74bd") pod "6215f0e0-d889-49ee-ab05-911e5eaebd74" (UID: "6215f0e0-d889-49ee-ab05-911e5eaebd74"). InnerVolumeSpecName "kube-api-access-s74bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:18:39.828126 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.828088 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6215f0e0-d889-49ee-ab05-911e5eaebd74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6215f0e0-d889-49ee-ab05-911e5eaebd74" (UID: "6215f0e0-d889-49ee-ab05-911e5eaebd74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:18:39.898046 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.898013 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74bd\" (UniqueName: \"kubernetes.io/projected/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-api-access-s74bd\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 18:18:39.898046 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.898044 2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6215f0e0-d889-49ee-ab05-911e5eaebd74-proxy-tls\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 18:18:39.898046 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.898053 2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6215f0e0-d889-49ee-ab05-911e5eaebd74-kserve-provision-location\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 18:18:39.898251 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.898062 2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 18:18:39.898251 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:39.898071 2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6215f0e0-d889-49ee-ab05-911e5eaebd74-cabundle-cert\") on node \"ip-10-0-134-110.ec2.internal\" DevicePath \"\"" Jul 28 18:18:40.269757 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.269719 2567 generic.go:356] "Generic (PLEG): container finished" podID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerID="0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c" exitCode=0 Jul 28 18:18:40.270221 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.269774 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" Jul 28 18:18:40.270221 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.269801 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerDied","Data":"0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c"} Jul 28 18:18:40.270221 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.269877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss" event={"ID":"6215f0e0-d889-49ee-ab05-911e5eaebd74","Type":"ContainerDied","Data":"7177847c30f27d6aaa69861420084aca647e823afcf4cbe93bb8915fe0f3721b"} Jul 28 18:18:40.270221 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.269897 2567 scope.go:117] "RemoveContainer" containerID="f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6" Jul 28 18:18:40.279028 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.279012 2567 scope.go:117] "RemoveContainer" containerID="0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c" Jul 28 18:18:40.286558 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.286540 2567 scope.go:117] "RemoveContainer" containerID="9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae" Jul 28 18:18:40.294118 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.294093 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss"] Jul 28 18:18:40.294814 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.294796 2567 scope.go:117] "RemoveContainer" containerID="f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6" Jul 28 18:18:40.295127 ip-10-0-134-110 kubenswrapper[2567]: E0728 18:18:40.295107 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6\": container with ID starting with f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6 not found: ID does not exist" containerID="f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6" Jul 28 18:18:40.295187 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.295136 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6"} err="failed to get container status \"f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6\": rpc error: code = NotFound desc = could not find container \"f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6\": container with ID starting with f49cf59c3b8d42a4d96075811febbc2fae497ca19542bf3e51220eced23b16f6 not found: ID does not exist" Jul 28 18:18:40.295187 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.295154 2567 scope.go:117] "RemoveContainer" containerID="0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c" Jul 28 18:18:40.295419 ip-10-0-134-110 kubenswrapper[2567]: E0728 18:18:40.295370 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c\": container with ID starting with 0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c not found: ID does not exist" containerID="0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c" Jul 28 18:18:40.295477 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.295426 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c"} err="failed to get container status \"0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c\": rpc error: code = NotFound desc = could not find container \"0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c\": container with ID starting with 0341958865be2dde0dfd52df47227ad9843b8fe776b9c0eddcbf8500d336ff5c not found: ID does not exist" Jul 28 18:18:40.295477 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.295441 2567 scope.go:117] "RemoveContainer" containerID="9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae" Jul 28 18:18:40.295742 ip-10-0-134-110 kubenswrapper[2567]: E0728 18:18:40.295714 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae\": container with ID starting with 9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae not found: ID does not exist" containerID="9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae" Jul 28 18:18:40.295856 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.295791 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae"} err="failed to get container status \"9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae\": rpc error: code = NotFound desc = could not find container \"9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae\": container with ID starting with 9144f3b5484ea632759d2e7e1a73cd00394f860ea266910e45cec41b2c6fbaae not found: ID does not exist" Jul 28 18:18:40.298444 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.298421 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-gfjss"] Jul 28 18:18:40.420596 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:18:40.420553 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" path="/var/lib/kubelet/pods/6215f0e0-d889-49ee-ab05-911e5eaebd74/volumes" Jul 28 18:41:33.791662 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.791598 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fsvbn/must-gather-ffgvk"] Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.791992 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="storage-initializer" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792005 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="storage-initializer" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792018 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792024 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792034 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kube-rbac-proxy" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792039 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kube-rbac-proxy" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792097 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kserve-container" Jul 28 18:41:33.794039 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.792103 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6215f0e0-d889-49ee-ab05-911e5eaebd74" containerName="kube-rbac-proxy" Jul 28 18:41:33.795058 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.795042 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:33.797924 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.797898 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fsvbn\"/\"openshift-service-ca.crt\"" Jul 28 18:41:33.798059 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.797935 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fsvbn\"/\"kube-root-ca.crt\"" Jul 28 18:41:33.799207 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.799187 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fsvbn\"/\"default-dockercfg-h49r6\"" Jul 28 18:41:33.803191 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.803168 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsvbn/must-gather-ffgvk"] Jul 28 18:41:33.815720 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.815690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b3222db-8e98-4694-9506-36960e2123c7-must-gather-output\") pod \"must-gather-ffgvk\" (UID: \"6b3222db-8e98-4694-9506-36960e2123c7\") " pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:33.815885 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.815822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgw9g\" (UniqueName: \"kubernetes.io/projected/6b3222db-8e98-4694-9506-36960e2123c7-kube-api-access-zgw9g\") pod \"must-gather-ffgvk\" (UID: \"6b3222db-8e98-4694-9506-36960e2123c7\") " pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:33.916444 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.916399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgw9g\" (UniqueName: \"kubernetes.io/projected/6b3222db-8e98-4694-9506-36960e2123c7-kube-api-access-zgw9g\") pod \"must-gather-ffgvk\" (UID: \"6b3222db-8e98-4694-9506-36960e2123c7\") " pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:33.916615 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.916476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b3222db-8e98-4694-9506-36960e2123c7-must-gather-output\") pod \"must-gather-ffgvk\" (UID: \"6b3222db-8e98-4694-9506-36960e2123c7\") " pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:33.916899 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.916853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b3222db-8e98-4694-9506-36960e2123c7-must-gather-output\") pod \"must-gather-ffgvk\" (UID: \"6b3222db-8e98-4694-9506-36960e2123c7\") " pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:33.925482 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:33.925450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgw9g\" (UniqueName: \"kubernetes.io/projected/6b3222db-8e98-4694-9506-36960e2123c7-kube-api-access-zgw9g\") pod \"must-gather-ffgvk\" (UID: \"6b3222db-8e98-4694-9506-36960e2123c7\") " pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:34.116351 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:34.116316 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsvbn/must-gather-ffgvk" Jul 28 18:41:34.247300 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:34.247261 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsvbn/must-gather-ffgvk"] Jul 28 18:41:34.249611 ip-10-0-134-110 kubenswrapper[2567]: W0728 18:41:34.249586 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b3222db_8e98_4694_9506_36960e2123c7.slice/crio-da6efaacbf532bd7381269a0358b4868b6503e3243bedf2144ac22f2c06b491a WatchSource:0}: Error finding container da6efaacbf532bd7381269a0358b4868b6503e3243bedf2144ac22f2c06b491a: Status 404 returned error can't find the container with id da6efaacbf532bd7381269a0358b4868b6503e3243bedf2144ac22f2c06b491a Jul 28 18:41:34.251620 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:34.251600 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:41:35.020793 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:35.020762 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/must-gather-ffgvk" event={"ID":"6b3222db-8e98-4694-9506-36960e2123c7","Type":"ContainerStarted","Data":"da6efaacbf532bd7381269a0358b4868b6503e3243bedf2144ac22f2c06b491a"} Jul 28 18:41:36.027876 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:36.027836 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/must-gather-ffgvk" event={"ID":"6b3222db-8e98-4694-9506-36960e2123c7","Type":"ContainerStarted","Data":"b450a403e1cbc51dc3e7857b1233ffd712dd7566e7aa30123a8dde248018adc3"} Jul 28 18:41:36.028285 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:36.027898 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/must-gather-ffgvk" event={"ID":"6b3222db-8e98-4694-9506-36960e2123c7","Type":"ContainerStarted","Data":"4420ceb2c35ed650edccebfc96e2af0c7ea45089dc23d749ee8e14cbf524c1f7"} Jul 28 18:41:36.046747 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:36.046684 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fsvbn/must-gather-ffgvk" podStartSLOduration=2.228783477 podStartE2EDuration="3.046666371s" totalImagesPullingTime="817.882894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:36.045957617 +0000 UTC m=+5538.111453584" watchObservedRunningTime="2026-07-28 18:41:36.046666371 +0000 UTC m=+5538.112162352" Jul 28 18:41:36.411782 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:36.411746 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2s7mz_e4997a7f-0501-47fb-ad19-35c54ca8d006/global-pull-secret-syncer/0.log" Jul 28 18:41:36.623933 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:36.623890 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n4s7j_6c7c6e25-8447-40b7-b2b1-52d9942add01/konnectivity-agent/0.log" Jul 28 18:41:36.716196 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:36.716112 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-110.ec2.internal_9005259af348bbf3d574131a972e70c3/haproxy/0.log" Jul 28 18:41:40.750510 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.750481 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/alertmanager/0.log" Jul 28 18:41:40.792100 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.792068 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/config-reloader/0.log" Jul 28 18:41:40.821908 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.821850 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/kube-rbac-proxy-web/0.log" Jul 28 18:41:40.843897 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.843814 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/kube-rbac-proxy/0.log" Jul 28 18:41:40.872353 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.872316 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/kube-rbac-proxy-metric/0.log" Jul 28 18:41:40.898694 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.898615 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/prom-label-proxy/0.log" Jul 28 18:41:40.922371 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:40.922345 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a7acbbb3-8e15-47db-8361-31f545deba15/init-config-reloader/0.log" Jul 28 18:41:41.018631 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.018552 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-s7x2k_9560b20a-d829-49f2-a693-f56c76a2297a/kube-state-metrics/0.log" Jul 28 18:41:41.058884 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.058839 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-s7x2k_9560b20a-d829-49f2-a693-f56c76a2297a/kube-rbac-proxy-main/0.log" Jul 28 18:41:41.083229 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.081800 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-s7x2k_9560b20a-d829-49f2-a693-f56c76a2297a/kube-rbac-proxy-self/0.log" Jul 28 18:41:41.113235 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.113202 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77b78d6768-nbdds_3387c220-7739-484c-abb4-32355494331f/metrics-server/0.log" Jul 28 18:41:41.142801 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.142767 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-m8q2k_93d4bef0-9df1-49ae-9352-59c92ea90981/monitoring-plugin/0.log" Jul 28 18:41:41.355613 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.355565 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wcwdt_a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5/node-exporter/0.log" Jul 28 18:41:41.384580 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.384470 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wcwdt_a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5/kube-rbac-proxy/0.log" Jul 28 18:41:41.409080 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.409052 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wcwdt_a6e59893-ed27-4dcc-87ec-eb5e46d9e8b5/init-textfile/0.log" Jul 28 18:41:41.442856 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.442822 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9zlg8_5af600dd-57af-45ad-9b7f-9bca4a252792/kube-rbac-proxy-main/0.log" Jul 28 18:41:41.470134 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.470096 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9zlg8_5af600dd-57af-45ad-9b7f-9bca4a252792/kube-rbac-proxy-self/0.log" Jul 28 18:41:41.501309 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.501281 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-9zlg8_5af600dd-57af-45ad-9b7f-9bca4a252792/openshift-state-metrics/0.log" Jul 28 18:41:41.558895 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.558848 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/prometheus/0.log" Jul 28 18:41:41.577164 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.577128 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/config-reloader/0.log" Jul 28 18:41:41.602101 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.602072 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/thanos-sidecar/0.log" Jul 28 18:41:41.628940 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.628839 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/kube-rbac-proxy-web/0.log" Jul 28 18:41:41.655758 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.655732 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/kube-rbac-proxy/0.log" Jul 28 18:41:41.682521 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.682492 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/kube-rbac-proxy-thanos/0.log" Jul 28 18:41:41.716207 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.716179 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_55c73256-263d-47f0-8f4b-dcdd4049e568/init-config-reloader/0.log" Jul 28 18:41:41.758317 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.758284 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-78klr_4d40ac35-ed57-4d1f-83b2-b61a850e9caa/prometheus-operator/0.log" Jul 28 18:41:41.773035 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.772998 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-78klr_4d40ac35-ed57-4d1f-83b2-b61a850e9caa/kube-rbac-proxy/0.log" Jul 28 18:41:41.796754 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.796720 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-5xqpj_366e1152-89b9-42a5-b134-e23a4ffb464f/prometheus-operator-admission-webhook/0.log" Jul 28 18:41:41.838886 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.838842 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dfff7fddf-gdf6n_eb004635-2806-43ea-af5b-28d931fe356f/telemeter-client/0.log" Jul 28 18:41:41.860800 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.860764 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dfff7fddf-gdf6n_eb004635-2806-43ea-af5b-28d931fe356f/reload/0.log" Jul 28 18:41:41.882849 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.882765 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dfff7fddf-gdf6n_eb004635-2806-43ea-af5b-28d931fe356f/kube-rbac-proxy/0.log" Jul 28 18:41:41.917078 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.917045 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d56d77cf6-wgvb7_ea4fcc8e-4446-47d8-8676-37679d510c18/thanos-query/0.log" Jul 28 18:41:41.948316 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.948288 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d56d77cf6-wgvb7_ea4fcc8e-4446-47d8-8676-37679d510c18/kube-rbac-proxy-web/0.log" Jul 28 18:41:41.973975 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.973938 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d56d77cf6-wgvb7_ea4fcc8e-4446-47d8-8676-37679d510c18/kube-rbac-proxy/0.log" Jul 28 18:41:41.998473 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:41.998275 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d56d77cf6-wgvb7_ea4fcc8e-4446-47d8-8676-37679d510c18/prom-label-proxy/0.log" Jul 28 18:41:42.023374 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:42.023343 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d56d77cf6-wgvb7_ea4fcc8e-4446-47d8-8676-37679d510c18/kube-rbac-proxy-rules/0.log" Jul 28 18:41:42.052306 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:42.052269 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d56d77cf6-wgvb7_ea4fcc8e-4446-47d8-8676-37679d510c18/kube-rbac-proxy-metrics/0.log" Jul 28 18:41:44.343082 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.343055 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b6f7fcf76-c2gv4_d124af8e-7cf9-4469-aa07-b2ed548c5620/console/0.log" Jul 28 18:41:44.737926 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.737889 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4"] Jul 28 18:41:44.743844 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.743817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.758109 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.758079 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4"] Jul 28 18:41:44.820984 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.820948 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-lib-modules\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.821162 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.821090 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-sys\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.821162 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.821146 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-podres\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.821241 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.821184 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-proc\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.821241 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.821214 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxl4c\" (UniqueName: \"kubernetes.io/projected/898b5821-c764-46c2-b8db-810421813f4e-kube-api-access-mxl4c\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.922801 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.922561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-lib-modules\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923037 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.922749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-lib-modules\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923037 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-sys\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923172 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-podres\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923172 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-sys\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923290 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-podres\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923290 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-proc\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923422 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxl4c\" (UniqueName: \"kubernetes.io/projected/898b5821-c764-46c2-b8db-810421813f4e-kube-api-access-mxl4c\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.923422 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.923399 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/898b5821-c764-46c2-b8db-810421813f4e-proc\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:44.933412 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:44.933380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxl4c\" (UniqueName: \"kubernetes.io/projected/898b5821-c764-46c2-b8db-810421813f4e-kube-api-access-mxl4c\") pod \"perf-node-gather-daemonset-xr8j4\" (UID: \"898b5821-c764-46c2-b8db-810421813f4e\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:45.057188 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:45.057100 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:45.223433 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:45.223406 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4"] Jul 28 18:41:45.225905 ip-10-0-134-110 kubenswrapper[2567]: W0728 18:41:45.225735 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod898b5821_c764_46c2_b8db_810421813f4e.slice/crio-4bd9d36ac05a8f8c3e665674617d8da35e3592d9ca241e36554814000cfad1fd WatchSource:0}: Error finding container 4bd9d36ac05a8f8c3e665674617d8da35e3592d9ca241e36554814000cfad1fd: Status 404 returned error can't find the container with id 4bd9d36ac05a8f8c3e665674617d8da35e3592d9ca241e36554814000cfad1fd Jul 28 18:41:45.828259 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:45.828167 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dp5lc_fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a/dns/0.log" Jul 28 18:41:45.849135 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:45.849093 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dp5lc_fe2cdbca-4cc3-4e78-bacc-9481f8fc9c5a/kube-rbac-proxy/0.log" Jul 28 18:41:45.990813 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:45.990787 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hfc7r_aef8329d-9fa4-4901-b7b4-8dda9ee78061/dns-node-resolver/0.log" Jul 28 18:41:46.090308 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:46.090212 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" event={"ID":"898b5821-c764-46c2-b8db-810421813f4e","Type":"ContainerStarted","Data":"2892668591da8f6a1c2b25671c014bd6c7dbc167e25bf55aab3457bb006c46db"} Jul 28 18:41:46.090308 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:46.090278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" event={"ID":"898b5821-c764-46c2-b8db-810421813f4e","Type":"ContainerStarted","Data":"4bd9d36ac05a8f8c3e665674617d8da35e3592d9ca241e36554814000cfad1fd"} Jul 28 18:41:46.090308 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:46.090306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:46.108273 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:46.108211 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" podStartSLOduration=2.108192187 podStartE2EDuration="2.108192187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:46.107619517 +0000 UTC m=+5548.173115486" watchObservedRunningTime="2026-07-28 18:41:46.108192187 +0000 UTC m=+5548.173688154" Jul 28 18:41:46.494519 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:46.494487 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78f6894798-z2wwk_52c988b9-2f97-47df-adf1-dda8d949890a/registry/0.log" Jul 28 18:41:46.570158 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:46.570128 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5g7t_00143af9-f792-4786-9bd2-aa61b5d1c32a/node-ca/0.log" Jul 28 18:41:47.600296 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:47.600258 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-622rh_5bb2f701-67fa-48f1-9afd-0c7ecd04039a/serve-healthcheck-canary/0.log" Jul 28 18:41:48.111293 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:48.111257 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zlnz9_bf7c933c-1dcc-4849-80b8-c5a349f19008/kube-rbac-proxy/0.log" Jul 28 18:41:48.130532 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:48.130503 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zlnz9_bf7c933c-1dcc-4849-80b8-c5a349f19008/exporter/0.log" Jul 28 18:41:48.151072 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:48.151022 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zlnz9_bf7c933c-1dcc-4849-80b8-c5a349f19008/extractor/0.log" Jul 28 18:41:51.163711 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:51.163680 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kwc47_e036a5e5-4f9f-42f4-ae4f-5b83037487fa/s3-init/0.log" Jul 28 18:41:51.178565 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:51.178533 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kwc47_e036a5e5-4f9f-42f4-ae4f-5b83037487fa/download-hf-model/0.log" Jul 28 18:41:51.211061 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:51.211012 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-cwl9t_05df1e67-595f-476d-9ac8-a93df2e2785e/s3-tls-init-custom/0.log" Jul 28 18:41:51.226132 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:51.226095 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-cwl9t_05df1e67-595f-476d-9ac8-a93df2e2785e/download-hf-model/0.log" Jul 28 18:41:51.259497 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:51.259454 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4nz22_9c3b58c0-81b7-43c8-90b0-b3a2f535b508/s3-tls-init-serving/0.log" Jul 28 18:41:51.275609 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:51.275563 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4nz22_9c3b58c0-81b7-43c8-90b0-b3a2f535b508/download-hf-model/0.log" Jul 28 18:41:52.105916 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:52.105856 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-xr8j4" Jul 28 18:41:56.738996 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.738914 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-46d7c_bc48a1bb-a5b6-4bc5-8c41-8fb2cb77726f/kube-multus/0.log" Jul 28 18:41:56.766437 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.766406 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/kube-multus-additional-cni-plugins/0.log" Jul 28 18:41:56.787506 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.787478 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/egress-router-binary-copy/0.log" Jul 28 18:41:56.807396 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.807369 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/cni-plugins/0.log" Jul 28 18:41:56.828753 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.828725 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/bond-cni-plugin/0.log" Jul 28 18:41:56.850665 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.850635 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/routeoverride-cni/0.log" Jul 28 18:41:56.871180 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.871151 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/whereabouts-cni-bincopy/0.log" Jul 28 18:41:56.893567 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:56.893538 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-44qsl_5351fc1f-c8b1-4f82-b5a1-90f8e1f2a924/whereabouts-cni/0.log" Jul 28 18:41:57.352731 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:57.352694 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nlxkb_acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5/network-metrics-daemon/0.log" Jul 28 18:41:57.373707 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:57.373676 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nlxkb_acfc9106-ddc9-4e2c-bdfe-a89314bf9fd5/kube-rbac-proxy/0.log" Jul 28 18:41:58.693757 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.693714 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/ovn-controller/0.log" Jul 28 18:41:58.710279 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.710253 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/ovn-acl-logging/0.log" Jul 28 18:41:58.732993 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.732964 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/kube-rbac-proxy-node/0.log" Jul 28 18:41:58.755646 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.755619 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 18:41:58.779809 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.779780 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/northd/0.log" Jul 28 18:41:58.806122 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.806079 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/nbdb/0.log" Jul 28 18:41:58.826614 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.826580 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/sbdb/0.log" Jul 28 18:41:58.893360 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:58.893326 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-swcbs_80259220-717f-491c-8b7d-1c499fac7d57/ovnkube-controller/0.log" Jul 28 18:41:59.809060 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:41:59.809032 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2bfzx_0923ada6-bef2-4608-a197-021abbcacd6d/network-check-target-container/0.log" Jul 28 18:42:00.731965 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:42:00.731933 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jx24d_fcc8c290-39e6-4f26-be39-9e67ad8ae399/iptables-alerter/0.log" Jul 28 18:42:01.365418 ip-10-0-134-110 kubenswrapper[2567]: I0728 18:42:01.365380 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-2tf6f_91c41323-027b-4d8e-a1f2-e81e190e4567/tuned/0.log"