Jul 19 14:55:55.621486 ip-10-0-142-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 19 14:55:55.621495 ip-10-0-142-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 19 14:55:55.621502 ip-10-0-142-31 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 19 14:55:55.621743 ip-10-0-142-31 systemd[1]: Failed to start Kubernetes Kubelet. Jul 19 14:56:05.779222 ip-10-0-142-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 19 14:56:05.779241 ip-10-0-142-31 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e7a347a029ab49c8bfd5aeafdde8749e -- Jul 19 14:57:57.018506 ip-10-0-142-31 systemd[1]: Starting Kubernetes Kubelet... Jul 19 14:57:57.396500 ip-10-0-142-31 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:57.396500 ip-10-0-142-31 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 19 14:57:57.396500 ip-10-0-142-31 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:57.396500 ip-10-0-142-31 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 19 14:57:57.396500 ip-10-0-142-31 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:57.399156 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.399070 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 19 14:57:57.403175 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403150 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403181 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403185 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403199 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403203 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403206 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403209 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403212 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403214 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403217 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403219 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403222 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403226 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:57.403222 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403229 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403232 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403243 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403246 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403248 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403253 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403256 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403259 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403262 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403265 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403268 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403271 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403274 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403276 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403279 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403282 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403284 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403287 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403290 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:57.403523 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403292 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403295 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403299 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403303 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403305 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403308 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403311 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403314 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403317 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403319 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403322 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403324 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403327 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403329 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403332 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403336 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403339 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403341 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403344 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403347 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:57.403951 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403349 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403352 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403354 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403357 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403359 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403362 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403365 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403368 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403370 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403373 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403376 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403378 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403381 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403384 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403386 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403389 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403392 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403395 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403397 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403400 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:57.404445 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403404 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403407 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403410 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403413 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403415 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403418 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403421 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403424 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403427 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403429 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403432 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403434 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403437 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403439 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403442 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403445 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403447 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403450 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403453 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403455 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:57.404910 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403458 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403461 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403463 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403467 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403470 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403474 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.403478 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404211 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404218 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404222 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404224 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404227 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404231 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404234 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404236 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404240 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404242 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404245 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404248 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404251 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:57.405410 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404254 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404257 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404259 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404262 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404265 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404267 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404270 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404273 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404275 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404278 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404280 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404283 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404286 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404289 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404292 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404294 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404297 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404299 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404302 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404305 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:57.405888 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404307 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404309 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404312 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404315 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404318 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404321 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404324 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404326 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404329 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404331 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404334 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404337 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404340 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404342 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404345 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404347 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404350 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404354 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404357 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404360 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:57.406360 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404363 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404365 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404370 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404374 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404377 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404382 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404385 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404388 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404391 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404394 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404397 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404399 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404402 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404405 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404407 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404410 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404413 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404416 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404419 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:57.406821 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404422 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404424 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404427 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404430 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404433 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404436 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404439 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404442 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404444 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404447 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404449 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404452 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404456 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404503 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404506 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404509 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404512 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404517 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404520 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:57.407305 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404523 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404526 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404529 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404532 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404535 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404539 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404542 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404545 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404925 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404931 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404934 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404937 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404944 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404947 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404949 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404952 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404955 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404959 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404962 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404964 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:57.407751 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404967 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404970 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404972 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404975 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404978 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404980 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404983 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404985 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404988 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404992 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404995 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.404998 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405001 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405004 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405007 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405009 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405012 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405015 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405017 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:57.408257 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405020 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405023 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405026 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405029 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405032 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405037 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405040 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405042 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405046 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405048 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405053 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405056 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405059 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405062 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405065 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405068 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405070 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405073 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:57.408700 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405075 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405078 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405080 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405083 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405085 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405088 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405091 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405093 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405096 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405099 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405101 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405104 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405106 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405109 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405112 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405114 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405116 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405119 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405121 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405123 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:57.409150 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405140 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405143 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405146 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405149 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405154 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405157 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405160 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405178 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405182 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405184 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405187 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405189 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405192 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405195 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405198 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405200 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405203 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405206 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405208 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:57.409667 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405211 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405214 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405216 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405219 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405221 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405224 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405226 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405229 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405231 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405233 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405306 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405312 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405319 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405323 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405328 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405333 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405338 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405342 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405345 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405349 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405352 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405356 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 19 14:57:57.410140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405359 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405362 2579 flags.go:64] FLAG: --cgroup-root="" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405365 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405368 2579 flags.go:64] FLAG: --client-ca-file="" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405371 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405374 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405378 2579 flags.go:64] FLAG: --cluster-domain="" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405381 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405384 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405387 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405390 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405394 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405397 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405400 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405403 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405407 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405409 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405412 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405416 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405419 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405423 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405426 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405429 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405432 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405435 2579 flags.go:64] FLAG: --enable-server="true" Jul 19 14:57:57.410702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405443 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405448 2579 flags.go:64] FLAG: --event-burst="100" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405451 2579 flags.go:64] FLAG: --event-qps="50" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405454 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405458 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405460 2579 flags.go:64] FLAG: --eviction-hard="" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405464 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405467 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405470 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405473 2579 flags.go:64] FLAG: --eviction-soft="" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405476 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405479 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405482 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405485 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405488 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405491 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405494 2579 flags.go:64] FLAG: --feature-gates="" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405498 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405501 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405504 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405508 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405511 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405514 2579 flags.go:64] FLAG: --help="false" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405516 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-31.ec2.internal" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405519 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 19 14:57:57.411301 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405522 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405525 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405528 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405532 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405534 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405537 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405542 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405546 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405550 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405553 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405556 2579 flags.go:64] FLAG: --kube-reserved="" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405559 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405561 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405565 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405567 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405570 2579 flags.go:64] FLAG: --lock-file="" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405573 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405576 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405579 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405584 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405587 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405590 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405593 2579 flags.go:64] FLAG: --logging-format="text" Jul 19 14:57:57.411906 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405596 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405599 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405602 2579 flags.go:64] FLAG: --manifest-url="" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405605 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405610 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405612 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405617 2579 flags.go:64] FLAG: --max-pods="110" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405620 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405622 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405625 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405628 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405631 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405634 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405637 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405645 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405649 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405653 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405656 2579 flags.go:64] FLAG: --pod-cidr="" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405660 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405665 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405668 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405671 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405674 2579 flags.go:64] FLAG: --port="10250" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405677 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 19 14:57:57.412484 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405681 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01be98e16cb854397" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405684 2579 flags.go:64] FLAG: --qos-reserved="" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405687 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405690 2579 flags.go:64] FLAG: --register-node="true" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405693 2579 flags.go:64] FLAG: --register-with-taints="" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405697 2579 flags.go:64] FLAG: --registry-burst="10" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405699 2579 flags.go:64] FLAG: --registry-qps="5" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405702 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405705 2579 flags.go:64] FLAG: --reserved-memory="" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405709 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405712 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405715 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405717 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405720 2579 flags.go:64] FLAG: --runonce="false" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405723 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405726 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405729 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405732 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405735 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405738 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405741 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405744 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405747 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405749 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405755 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405758 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 19 14:57:57.413076 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405761 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405764 2579 flags.go:64] FLAG: --system-cgroups="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405767 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405772 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405775 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405778 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405782 2579 flags.go:64] FLAG: --tls-min-version="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405784 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405787 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405790 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405793 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405796 2579 flags.go:64] FLAG: --v="2" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405799 2579 flags.go:64] FLAG: --version="false" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405803 2579 flags.go:64] FLAG: --vmodule="" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405807 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.405810 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405920 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405923 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405926 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405929 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405933 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405936 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405939 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:57.413701 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405942 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405945 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405947 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405950 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405953 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405955 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405958 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405963 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405966 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405969 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405971 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405975 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405977 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405980 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405982 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405985 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405987 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405990 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405994 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:57.414293 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.405998 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406001 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406003 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406006 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406008 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406011 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406013 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406016 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406019 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406021 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406025 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406029 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406031 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406037 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406039 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406042 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406045 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406047 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406050 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:57.414758 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406055 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406058 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406060 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406063 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406065 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406068 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406072 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406074 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406077 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406079 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406082 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406085 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406088 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406090 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406093 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406095 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406098 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406101 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406103 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406106 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:57.415302 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406108 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406111 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406114 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406116 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406119 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406121 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406124 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406126 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406129 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406131 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406134 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406137 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406144 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406147 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406150 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406153 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406155 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406158 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406174 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:57.415772 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406177 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406180 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406183 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406186 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406188 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406191 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406194 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406196 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406199 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406202 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406205 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406207 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406210 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.406213 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.406219 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:57.416230 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.413524 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.413538 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.413569 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.413576 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413632 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413637 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413641 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413644 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413648 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413652 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413655 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413658 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413661 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413664 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413667 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413670 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413673 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413676 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413678 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:57.416603 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413681 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413684 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413686 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413689 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413691 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413694 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413697 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413700 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413702 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413705 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413708 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413711 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413713 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413716 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413720 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413722 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413725 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413728 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413730 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413733 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:57.417067 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413735 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413738 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413740 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413743 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413747 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413751 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413754 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413757 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413760 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413763 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413766 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413769 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413771 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413774 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413777 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413779 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413782 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413785 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413787 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:57.417569 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413790 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413793 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413795 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413798 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413801 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413803 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413806 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413809 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413813 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413815 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413818 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413821 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413823 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413826 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413828 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413831 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413834 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413836 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413839 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413841 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:57.418014 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413844 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413847 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413850 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413852 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413856 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413859 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413861 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413864 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413868 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413871 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413874 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413876 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413879 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413881 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413885 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413889 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413892 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413894 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413897 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:57.418491 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413900 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413903 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413906 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413908 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413911 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.413913 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.413918 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414031 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414036 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414039 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414042 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414045 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414048 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414052 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:57.418946 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414055 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414059 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414062 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414065 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414068 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414071 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414074 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414077 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414080 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414082 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414085 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414088 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414091 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414094 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414097 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414099 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414102 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414104 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414108 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414110 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:57.419326 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414113 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414115 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414118 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414121 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414123 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414126 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414128 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414131 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414135 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414138 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414141 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414144 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414146 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414149 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414152 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414155 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414157 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414160 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414178 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:57.419796 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414181 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414184 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414186 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414189 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414192 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414195 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414197 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414200 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414202 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414205 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414208 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414211 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414213 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414216 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414219 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414221 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414224 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414227 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414229 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414232 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:57.420266 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414234 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414237 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414239 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414242 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414244 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414247 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414250 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414252 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414255 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414257 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414260 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414263 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414266 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414268 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414271 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414273 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414276 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414278 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414281 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:57.420746 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414285 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414288 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414290 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414293 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414296 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414299 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414302 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414304 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414307 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414310 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414312 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414315 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414317 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:57.414320 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.414325 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:57.421210 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.414430 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 19 14:57:57.421582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.416945 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 19 14:57:57.421582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.418225 2579 server.go:1019] "Starting client certificate rotation" Jul 19 14:57:57.421582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.418321 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:57:57.421582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.419048 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:57:57.421582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.420442 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:57:57.423055 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.421573 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:57:57.431553 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.431487 2579 log.go:25] "Validated CRI v1 runtime API" Jul 19 14:57:57.433490 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.433473 2579 log.go:25] "Validated CRI v1 image API" Jul 19 14:57:57.434304 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.434288 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 19 14:57:57.436574 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.436549 2579 fs.go:135] Filesystem UUIDs: map[71a2f7c7-cb16-471e-993c-da4d591344cc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 82d7be8f-0b6f-482a-87a7-741740f44425:/dev/nvme0n1p3] Jul 19 14:57:57.436632 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.436574 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 19 14:57:57.442731 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.442618 2579 manager.go:216] Machine: {Timestamp:2026-07-19 14:57:57.441414137 +0000 UTC m=+0.309062314 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096666 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230b355bccac62f6db6be9ffc56e76 SystemUUID:ec230b35-5bcc-ac62-f6db-6be9ffc56e76 BootID:e7a347a0-29ab-49c8-bfd5-aeafdde8749e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5c:0b:c8:6c:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5c:0b:c8:6c:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:c0:84:8d:84:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 19 14:57:57.442731 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.442728 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 19 14:57:57.442850 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.442809 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 19 14:57:57.446027 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.445999 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 19 14:57:57.446230 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.446030 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 19 14:57:57.447235 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.447223 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 19 14:57:57.447275 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.447238 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 19 14:57:57.447275 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.447246 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:57:57.447275 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.447260 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:57:57.448449 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.448436 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 19 14:57:57.451378 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451366 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:57:57.451528 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451517 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 19 14:57:57.451566 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451556 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 19 14:57:57.451602 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451577 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 19 14:57:57.451602 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451596 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 19 14:57:57.451653 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451609 2579 kubelet.go:398] "Adding apiserver pod source" Jul 19 14:57:57.451653 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451620 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 19 14:57:57.451653 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451624 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:57:57.452001 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.451988 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:57:57.452069 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.452006 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:57:57.453908 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.453886 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 19 14:57:57.455983 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.455955 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 19 14:57:57.456080 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456007 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 19 14:57:57.456880 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456861 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 19 14:57:57.456880 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456883 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 19 14:57:57.456880 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456890 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 19 14:57:57.456880 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456895 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456904 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456910 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456918 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456923 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456930 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456935 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456944 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 19 14:57:57.457113 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.456949 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 19 14:57:57.457593 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.457583 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 19 14:57:57.457593 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.457592 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 19 14:57:57.460275 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.460252 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 19 14:57:57.460398 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.460342 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 19 14:57:57.460465 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.460342 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 19 14:57:57.460844 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.460831 2579 server.go:1268] "Started kubelet" Jul 19 14:57:57.460937 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.460909 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 19 14:57:57.461675 ip-10-0-142-31 systemd[1]: Started Kubernetes Kubelet. Jul 19 14:57:57.461815 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.461630 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 19 14:57:57.461815 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.461708 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 19 14:57:57.463177 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.463146 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 19 14:57:57.463521 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.463509 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 19 14:57:57.465598 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.464836 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-31.ec2.internal.18c3b8cf354c9ae1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-31.ec2.internal,UID:ip-10-0-142-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-31.ec2.internal,},FirstTimestamp:2026-07-19 14:57:57.460802273 +0000 UTC m=+0.328450451,LastTimestamp:2026-07-19 14:57:57.460802273 +0000 UTC m=+0.328450451,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-31.ec2.internal,}" Jul 19 14:57:57.467035 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.467016 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 19 14:57:57.469659 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.469640 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 19 14:57:57.469726 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.469654 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 19 14:57:57.469773 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.469730 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:57.470137 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470121 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 19 14:57:57.470278 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470123 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 19 14:57:57.470361 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470281 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 19 14:57:57.470419 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470394 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 19 14:57:57.470419 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470400 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 19 14:57:57.470769 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470754 2579 factory.go:153] Registering CRI-O factory Jul 19 14:57:57.470820 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470774 2579 factory.go:223] Registration of the crio container factory successfully Jul 19 14:57:57.470858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470836 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 19 14:57:57.470858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470845 2579 factory.go:55] Registering systemd factory Jul 19 14:57:57.470858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470852 2579 factory.go:223] Registration of the systemd container factory successfully Jul 19 14:57:57.470961 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470877 2579 factory.go:103] Registering Raw factory Jul 19 14:57:57.470961 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.470892 2579 manager.go:1196] Started watching for new ooms in manager Jul 19 14:57:57.471348 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.471334 2579 manager.go:349] Starting recovery of all containers Jul 19 14:57:57.471533 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.471516 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 19 14:57:57.471611 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.471595 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 19 14:57:57.472516 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.472442 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2hjzv" Jul 19 14:57:57.479993 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.479971 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2hjzv" Jul 19 14:57:57.483125 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.483107 2579 manager.go:354] Recovery completed Jul 19 14:57:57.485434 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.485413 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 19 14:57:57.488421 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.488408 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:57.490372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490357 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:57.490438 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490383 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:57.490438 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490394 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:57.490850 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490837 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 19 14:57:57.490850 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490850 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 19 14:57:57.490953 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490861 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:57.490953 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.490865 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:57:57.491591 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.491573 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:57.491655 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.491600 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:57.491655 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.491610 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:57.492073 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.492057 2579 policy_none.go:49] "None policy: Start" Jul 19 14:57:57.492073 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.492074 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 19 14:57:57.492207 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.492084 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 19 14:57:57.493127 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.493115 2579 policy_none.go:47] "Start" Jul 19 14:57:57.524768 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.524746 2579 manager.go:337] "Starting Device Plugin manager" Jul 19 14:57:57.524873 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.524783 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 19 14:57:57.524873 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.524797 2579 server.go:87] "Starting device plugin registration server" Jul 19 14:57:57.525147 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.525027 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 19 14:57:57.525213 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.525153 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 19 14:57:57.525263 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.525234 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 19 14:57:57.525324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.525312 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 19 14:57:57.525366 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.525329 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 19 14:57:57.525837 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.525813 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 19 14:57:57.525914 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.525869 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:57.598181 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.598132 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 19 14:57:57.599323 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.599300 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 19 14:57:57.599383 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.599337 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 19 14:57:57.599383 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.599361 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 19 14:57:57.599465 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.599393 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 19 14:57:57.601383 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.601362 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:57.626308 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.626286 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:57.627141 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.627124 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:57.627248 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.627155 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:57.627248 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.627180 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:57.627248 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.627212 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-31.ec2.internal" Jul 19 14:57:57.634380 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:57.634365 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-31.ec2.internal" Jul 19 14:57:57.634464 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.634391 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-31.ec2.internal\": node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:57.647298 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:57.647276 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:58.707530 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.707507 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"] Jul 19 14:57:58.707598 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.707585 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:58.708450 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.708433 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:58.708542 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.708467 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:58.708542 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.708481 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:58.709850 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.709836 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:58.709996 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.709980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.710039 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710014 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:58.710597 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710579 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:58.710680 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710601 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:58.710680 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710611 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:58.710680 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710618 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:58.710680 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710638 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:58.710680 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.710662 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:58.711676 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.711659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.711759 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.711684 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:58.712280 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.712249 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:58.712349 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.712292 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:58.712349 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.712307 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:58.735935 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:58.735910 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-31.ec2.internal\" not found" node="ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.740531 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:58.740511 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-31.ec2.internal\" not found" node="ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.754605 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:58.754582 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:58.779813 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.779791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/792c18f67f33440108ee3acdda7b409b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"792c18f67f33440108ee3acdda7b409b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.779896 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.779818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a290a0755b8688c1e5c83007894063ed-config\") pod \"kube-apiserver-proxy-ip-10-0-142-31.ec2.internal\" (UID: \"a290a0755b8688c1e5c83007894063ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.779896 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.779835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/792c18f67f33440108ee3acdda7b409b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"792c18f67f33440108ee3acdda7b409b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.854804 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:58.854776 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:58.880130 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.880107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/792c18f67f33440108ee3acdda7b409b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"792c18f67f33440108ee3acdda7b409b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.880190 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.880133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/792c18f67f33440108ee3acdda7b409b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"792c18f67f33440108ee3acdda7b409b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.880190 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.880149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a290a0755b8688c1e5c83007894063ed-config\") pod \"kube-apiserver-proxy-ip-10-0-142-31.ec2.internal\" (UID: \"a290a0755b8688c1e5c83007894063ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.880190 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.880176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a290a0755b8688c1e5c83007894063ed-config\") pod \"kube-apiserver-proxy-ip-10-0-142-31.ec2.internal\" (UID: \"a290a0755b8688c1e5c83007894063ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.880296 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.880216 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/792c18f67f33440108ee3acdda7b409b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"792c18f67f33440108ee3acdda7b409b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.880296 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:58.880224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/792c18f67f33440108ee3acdda7b409b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"792c18f67f33440108ee3acdda7b409b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:58.955460 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:58.955376 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:59.038084 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.038049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:59.044140 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.044123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" Jul 19 14:57:59.055874 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.055851 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:59.156433 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.156399 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:59.256946 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.256871 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:59.357536 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.357499 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found" Jul 19 14:57:59.425008 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.424976 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 19 14:57:59.425497 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.425172 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:59.444943 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.444920 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:59.458985 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.458959 2579 apiserver.go:52] "Watching apiserver" Jul 19 14:57:59.468584 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.468558 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 19 14:57:59.469547 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.469520 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mg8z6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2","openshift-image-registry/node-ca-x666x","openshift-multus/multus-additional-cni-plugins-8kfl7","openshift-multus/network-metrics-daemon-tc8s7","openshift-network-diagnostics/network-check-target-5jldc","openshift-ovn-kubernetes/ovnkube-node-vzhcm","kube-system/konnectivity-agent-vshh6","openshift-cluster-node-tuning-operator/tuned-lhq55","openshift-dns/node-resolver-bc55h","openshift-multus/multus-9g5fl","openshift-network-operator/iptables-alerter-x6qj6"] Jul 19 14:57:59.471004 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.470984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.471094 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.471054 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:57:59.472530 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.472509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.476416 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.475701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.476416 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.475966 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.476416 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.476001 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxwfw\"" Jul 19 14:57:59.476416 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.476184 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.476416 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.476416 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 19 14:57:59.476870 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.476852 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 19 14:57:59.476934 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.476892 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" Jul 19 14:57:59.477543 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.477089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.478582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.478562 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.478700 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.478683 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 19 14:57:59.478752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.478705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:57:59.478825 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.478788 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:57:59.478957 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.478942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.479088 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.479073 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qknzm\"" Jul 19 14:57:59.479528 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.479507 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.479615 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.479531 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.479909 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.479894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 19 14:57:59.480006 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.479992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 19 14:57:59.480173 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.480160 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 19 14:57:59.480211 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.480176 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qntgh\"" Jul 19 14:57:59.480587 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.480570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:57:59.480666 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.480632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:57:59.482368 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.482351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.483532 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-os-release\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483607 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483607 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z2s\" (UniqueName: \"kubernetes.io/projected/8717acab-00c9-46d6-b88b-fcc123b04b9c-kube-api-access-26z2s\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.483607 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.483607 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-system-cni-dir\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-cnibin\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-dbus\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.483752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.483752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-socket-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.483752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-registration-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.483752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnf8\" (UniqueName: \"kubernetes.io/projected/099f1352-0492-4608-ba23-0a6b4bb8ac0b-kube-api-access-mtnf8\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.483974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8717acab-00c9-46d6-b88b-fcc123b04b9c-host\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.483974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxks\" (UniqueName: \"kubernetes.io/projected/69db3133-e92e-47d0-9a7b-38d6501694b9-kube-api-access-wpxks\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.483974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.483990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-sys-fs\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8717acab-00c9-46d6-b88b-fcc123b04b9c-serviceca\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ng7\" (UniqueName: \"kubernetes.io/projected/b9f28b58-145b-4dcb-a3b2-82741e2ad378-kube-api-access-d8ng7\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-kubelet-config\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.484134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-device-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.484685 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.484672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.485081 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485068 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.485155 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485105 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 19 14:57:59.485228 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzdkc\"" Jul 19 14:57:59.485228 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485172 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 19 14:57:59.485228 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485191 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 19 14:57:59.485417 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485299 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.485660 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485646 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 19 14:57:59.485928 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.485914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.487032 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.487018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bq7zq\"" Jul 19 14:57:59.487229 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.487214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.487349 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.487332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 19 14:57:59.487417 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.487403 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 19 14:57:59.488324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.488308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.489023 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.489009 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xvqbp\"" Jul 19 14:57:59.489221 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.489205 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.489426 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.489412 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.489688 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.489675 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.490092 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.490077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.490509 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.490487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 19 14:57:59.490597 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.490542 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2z896\"" Jul 19 14:57:59.490662 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.490597 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4c9qp\"" Jul 19 14:57:59.490853 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.490838 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 19 14:57:59.491735 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.491719 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-59kt9\"" Jul 19 14:57:59.491806 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.491747 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:57:59.492052 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.492038 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:59.492501 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.492485 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 19 14:57:59.492855 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.492839 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:57:59.493486 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.493470 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:57:59.493486 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.493482 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"] Jul 19 14:57:59.493581 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.493530 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" Jul 19 14:57:59.494415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.494375 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 14:52:58 +0000 UTC" deadline="2028-01-08 18:28:26.894016521 +0000 UTC" Jul 19 14:57:59.494488 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.494415 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12915h30m27.399605373s" Jul 19 14:57:59.501500 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.501482 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"] Jul 19 14:57:59.501625 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.501604 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:57:59.512249 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.512231 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zq7lf" Jul 19 14:57:59.517808 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.517791 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zq7lf" Jul 19 14:57:59.568015 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.567978 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792c18f67f33440108ee3acdda7b409b.slice/crio-b96adb41707a233d18e69745b94679a139387a4b926b7db632f6c91a0903412e WatchSource:0}: Error finding container b96adb41707a233d18e69745b94679a139387a4b926b7db632f6c91a0903412e: Status 404 returned error can't find the container with id b96adb41707a233d18e69745b94679a139387a4b926b7db632f6c91a0903412e Jul 19 14:57:59.568232 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.568214 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda290a0755b8688c1e5c83007894063ed.slice/crio-6d06635b3f0beafed96c1985b3a21406b99aa33a019b9c184a89eaa6adccd7d8 WatchSource:0}: Error finding container 6d06635b3f0beafed96c1985b3a21406b99aa33a019b9c184a89eaa6adccd7d8: Status 404 returned error can't find the container with id 6d06635b3f0beafed96c1985b3a21406b99aa33a019b9c184a89eaa6adccd7d8 Jul 19 14:57:59.571210 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.571188 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 14:57:59.578222 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.578206 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 19 14:57:59.582473 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.582456 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:59.584317 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovnkube-config\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.584415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-env-overrides\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.584415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovnkube-script-lib\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.584415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8717acab-00c9-46d6-b88b-fcc123b04b9c-serviceca\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.584415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-log-socket\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.584415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-os-release\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-device-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-run\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.584512 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-hosts-file\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-device-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-cni-bin\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-hostroot\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.584661 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.584630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:00.0846127 +0000 UTC m=+1.945168682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-modprobe-d\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z2s\" (UniqueName: \"kubernetes.io/projected/8717acab-00c9-46d6-b88b-fcc123b04b9c-kube-api-access-26z2s\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d969b-9bc6-4405-8067-f503b71bf9d0-konnectivity-ca\") pod \"konnectivity-agent-vshh6\" (UID: \"6f8d969b-9bc6-4405-8067-f503b71bf9d0\") " pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-cnibin\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-dbus\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-socket-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-cnibin\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8717acab-00c9-46d6-b88b-fcc123b04b9c-host\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-slash\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-socket-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8717acab-00c9-46d6-b88b-fcc123b04b9c-host\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53d7f044-9eb2-4dc3-8770-47651099e3d3-host-slash\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584980 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-dbus\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.584984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e006bb-0ec5-41ca-a29b-ee609c919dfd-tmp\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.585144 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-k8s-cni-cncf-io\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-netns\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-conf-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53d7f044-9eb2-4dc3-8770-47651099e3d3-iptables-alerter-script\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-kubernetes\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-sys\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-tuned\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-os-release\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwst\" (UniqueName: \"kubernetes.io/projected/06e006bb-0ec5-41ca-a29b-ee609c919dfd-kube-api-access-ttwst\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-cni-bin\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8717acab-00c9-46d6-b88b-fcc123b04b9c-serviceca\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/172caf8f-fd5d-4508-962d-ee98378bf734-multus-daemon-config\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-os-release\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-system-cni-dir\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.585918 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69db3133-e92e-47d0-9a7b-38d6501694b9-system-cni-dir\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-registration-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585621 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-registration-dir\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6hd\" (UniqueName: \"kubernetes.io/projected/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-kube-api-access-rf6hd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-kubelet\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-multus-certs\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-tmp-dir\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-cnibin\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovn-node-metrics-cert\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfzq\" (UniqueName: \"kubernetes.io/projected/53d7f044-9eb2-4dc3-8770-47651099e3d3-kube-api-access-8hfzq\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-systemd\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-systemd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.586663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-var-lib-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.585990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-cni-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnf8\" (UniqueName: \"kubernetes.io/projected/099f1352-0492-4608-ba23-0a6b4bb8ac0b-kube-api-access-mtnf8\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysctl-conf\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-kubelet\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-systemd-units\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxks\" (UniqueName: \"kubernetes.io/projected/69db3133-e92e-47d0-9a7b-38d6501694b9-kube-api-access-wpxks\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-sys-fs\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysconfig\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-etc-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/099f1352-0492-4608-ba23-0a6b4bb8ac0b-sys-fs\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-system-cni-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.586402 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586485 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586646 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.587251 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-socket-dir-parent\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.586813 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:00.086795122 +0000 UTC m=+1.947351088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ng7\" (UniqueName: \"kubernetes.io/projected/b9f28b58-145b-4dcb-a3b2-82741e2ad378-kube-api-access-d8ng7\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-kubelet-config\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysctl-d\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/172caf8f-fd5d-4508-962d-ee98378bf734-cni-binary-copy\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-cni-multus\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.586997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-kubelet-config\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-etc-kubernetes\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvfn\" (UniqueName: \"kubernetes.io/projected/172caf8f-fd5d-4508-962d-ee98378bf734-kube-api-access-fwvfn\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-node-log\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-var-lib-kubelet\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-host\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-run-netns\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-ovn\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.587793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f8d969b-9bc6-4405-8067-f503b71bf9d0-agent-certs\") pod \"konnectivity-agent-vshh6\" (UID: \"6f8d969b-9bc6-4405-8067-f503b71bf9d0\") " pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.588240 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69db3133-e92e-47d0-9a7b-38d6501694b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.588240 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-lib-modules\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.588240 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jg5\" (UniqueName: \"kubernetes.io/projected/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-kube-api-access-42jg5\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.588240 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.587295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-cni-netd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.591195 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.591175 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 19 14:57:59.594773 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.594753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ng7\" (UniqueName: \"kubernetes.io/projected/b9f28b58-145b-4dcb-a3b2-82741e2ad378-kube-api-access-d8ng7\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:57:59.594773 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.594763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxks\" (UniqueName: \"kubernetes.io/projected/69db3133-e92e-47d0-9a7b-38d6501694b9-kube-api-access-wpxks\") pod \"multus-additional-cni-plugins-8kfl7\" (UID: \"69db3133-e92e-47d0-9a7b-38d6501694b9\") " pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.594895 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.594756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z2s\" (UniqueName: \"kubernetes.io/projected/8717acab-00c9-46d6-b88b-fcc123b04b9c-kube-api-access-26z2s\") pod \"node-ca-x666x\" (UID: \"8717acab-00c9-46d6-b88b-fcc123b04b9c\") " pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.594895 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.594840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnf8\" (UniqueName: \"kubernetes.io/projected/099f1352-0492-4608-ba23-0a6b4bb8ac0b-kube-api-access-mtnf8\") pod \"aws-ebs-csi-driver-node-l7xj2\" (UID: \"099f1352-0492-4608-ba23-0a6b4bb8ac0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.610627 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.610586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" event={"ID":"792c18f67f33440108ee3acdda7b409b","Type":"ContainerStarted","Data":"b96adb41707a233d18e69745b94679a139387a4b926b7db632f6c91a0903412e"} Jul 19 14:57:59.611534 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.611514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" event={"ID":"a290a0755b8688c1e5c83007894063ed","Type":"ContainerStarted","Data":"6d06635b3f0beafed96c1985b3a21406b99aa33a019b9c184a89eaa6adccd7d8"} Jul 19 14:57:59.687762 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysconfig\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.687762 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-etc-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-system-cni-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-socket-dir-parent\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysctl-d\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/172caf8f-fd5d-4508-962d-ee98378bf734-cni-binary-copy\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-etc-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-cni-multus\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysconfig\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-etc-kubernetes\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-socket-dir-parent\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-cni-multus\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvfn\" (UniqueName: \"kubernetes.io/projected/172caf8f-fd5d-4508-962d-ee98378bf734-kube-api-access-fwvfn\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-system-cni-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-etc-kubernetes\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-node-log\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-node-log\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.687989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-var-lib-kubelet\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysctl-d\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.687999 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-host\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-run-netns\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-var-lib-kubelet\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-ovn\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-host\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-run-netns\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-ovn\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f8d969b-9bc6-4405-8067-f503b71bf9d0-agent-certs\") pod \"konnectivity-agent-vshh6\" (UID: \"6f8d969b-9bc6-4405-8067-f503b71bf9d0\") " pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-lib-modules\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jg5\" (UniqueName: \"kubernetes.io/projected/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-kube-api-access-42jg5\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-cni-netd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688206 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-cni-netd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-lib-modules\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovnkube-config\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-env-overrides\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovnkube-script-lib\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-log-socket\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.688858 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-os-release\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-run\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/172caf8f-fd5d-4508-962d-ee98378bf734-cni-binary-copy\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-log-socket\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-hosts-file\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-run\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-os-release\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-hosts-file\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-cni-bin\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-hostroot\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-modprobe-d\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-hostroot\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-cni-bin\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d969b-9bc6-4405-8067-f503b71bf9d0-konnectivity-ca\") pod \"konnectivity-agent-vshh6\" (UID: \"6f8d969b-9bc6-4405-8067-f503b71bf9d0\") " pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-modprobe-d\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.689705 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-slash\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53d7f044-9eb2-4dc3-8770-47651099e3d3-host-slash\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e006bb-0ec5-41ca-a29b-ee609c919dfd-tmp\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-k8s-cni-cncf-io\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovnkube-config\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-slash\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53d7f044-9eb2-4dc3-8770-47651099e3d3-host-slash\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-netns\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-conf-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-k8s-cni-cncf-io\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53d7f044-9eb2-4dc3-8770-47651099e3d3-iptables-alerter-script\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-netns\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.688982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-conf-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-kubernetes\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovnkube-script-lib\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-sys\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.690468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-tuned\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-kubernetes\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwst\" (UniqueName: \"kubernetes.io/projected/06e006bb-0ec5-41ca-a29b-ee609c919dfd-kube-api-access-ttwst\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-cni-bin\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/172caf8f-fd5d-4508-962d-ee98378bf734-multus-daemon-config\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689310 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-cni-bin\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6hd\" (UniqueName: \"kubernetes.io/projected/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-kube-api-access-rf6hd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-kubelet\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-multus-certs\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-env-overrides\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-tmp-dir\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-cnibin\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovn-node-metrics-cert\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53d7f044-9eb2-4dc3-8770-47651099e3d3-iptables-alerter-script\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfzq\" (UniqueName: \"kubernetes.io/projected/53d7f044-9eb2-4dc3-8770-47651099e3d3-kube-api-access-8hfzq\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-systemd\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691020 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-tmp-dir\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-var-lib-kubelet\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-host-run-multus-certs\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-cnibin\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/172caf8f-fd5d-4508-962d-ee98378bf734-multus-daemon-config\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.689161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-sys\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-systemd\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d969b-9bc6-4405-8067-f503b71bf9d0-konnectivity-ca\") pod \"konnectivity-agent-vshh6\" (UID: \"6f8d969b-9bc6-4405-8067-f503b71bf9d0\") " pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-systemd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-run-systemd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-var-lib-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-cni-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-var-lib-openvswitch\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysctl-conf\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-kubelet\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-systemd-units\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/172caf8f-fd5d-4508-962d-ee98378bf734-multus-cni-dir\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-host-kubelet\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.691538 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-systemd-units\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.692086 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-sysctl-conf\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.692086 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.690794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f8d969b-9bc6-4405-8067-f503b71bf9d0-agent-certs\") pod \"konnectivity-agent-vshh6\" (UID: \"6f8d969b-9bc6-4405-8067-f503b71bf9d0\") " pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.692086 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.691452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06e006bb-0ec5-41ca-a29b-ee609c919dfd-tmp\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.692086 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.691656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06e006bb-0ec5-41ca-a29b-ee609c919dfd-etc-tuned\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.692086 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.691948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-ovn-node-metrics-cert\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.695166 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.695136 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:59.695166 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.695165 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:59.695325 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.695179 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:59.695325 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:57:59.695235 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:00.19521724 +0000 UTC m=+2.055773225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:59.696831 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.696806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvfn\" (UniqueName: \"kubernetes.io/projected/172caf8f-fd5d-4508-962d-ee98378bf734-kube-api-access-fwvfn\") pod \"multus-9g5fl\" (UID: \"172caf8f-fd5d-4508-962d-ee98378bf734\") " pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.696913 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.696851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jg5\" (UniqueName: \"kubernetes.io/projected/2ca150b8-5079-4fb5-99b1-4133ba4ae43a-kube-api-access-42jg5\") pod \"node-resolver-bc55h\" (UID: \"2ca150b8-5079-4fb5-99b1-4133ba4ae43a\") " pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.697218 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.697198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwst\" (UniqueName: \"kubernetes.io/projected/06e006bb-0ec5-41ca-a29b-ee609c919dfd-kube-api-access-ttwst\") pod \"tuned-lhq55\" (UID: \"06e006bb-0ec5-41ca-a29b-ee609c919dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.697537 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.697523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfzq\" (UniqueName: \"kubernetes.io/projected/53d7f044-9eb2-4dc3-8770-47651099e3d3-kube-api-access-8hfzq\") pod \"iptables-alerter-x6qj6\" (UID: \"53d7f044-9eb2-4dc3-8770-47651099e3d3\") " pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.697709 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.697694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6hd\" (UniqueName: \"kubernetes.io/projected/0b7adfdb-fcc8-4038-8660-cfe064a9c5a7-kube-api-access-rf6hd\") pod \"ovnkube-node-vzhcm\" (UID: \"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.808706 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.808624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" Jul 19 14:57:59.814576 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.814532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x666x" Jul 19 14:57:59.816782 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.816760 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099f1352_0492_4608_ba23_0a6b4bb8ac0b.slice/crio-17a38cde8a9765ce16230c92731428a5f6ac3b28847478b22882876f4311c1d9 WatchSource:0}: Error finding container 17a38cde8a9765ce16230c92731428a5f6ac3b28847478b22882876f4311c1d9: Status 404 returned error can't find the container with id 17a38cde8a9765ce16230c92731428a5f6ac3b28847478b22882876f4311c1d9 Jul 19 14:57:59.821679 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.821657 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8717acab_00c9_46d6_b88b_fcc123b04b9c.slice/crio-22a13368e7607b31617fce3b21134c2f5bf35f2a8498e42b3b228c5b028f9d8d WatchSource:0}: Error finding container 22a13368e7607b31617fce3b21134c2f5bf35f2a8498e42b3b228c5b028f9d8d: Status 404 returned error can't find the container with id 22a13368e7607b31617fce3b21134c2f5bf35f2a8498e42b3b228c5b028f9d8d Jul 19 14:57:59.830325 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.830269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" Jul 19 14:57:59.837422 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.837403 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69db3133_e92e_47d0_9a7b_38d6501694b9.slice/crio-7e1f4d5d33ff321e1bf707056f6decbd2ffd2fbf8902dad0a917bf3a52317c33 WatchSource:0}: Error finding container 7e1f4d5d33ff321e1bf707056f6decbd2ffd2fbf8902dad0a917bf3a52317c33: Status 404 returned error can't find the container with id 7e1f4d5d33ff321e1bf707056f6decbd2ffd2fbf8902dad0a917bf3a52317c33 Jul 19 14:57:59.848643 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.848625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:57:59.855614 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.855592 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7adfdb_fcc8_4038_8660_cfe064a9c5a7.slice/crio-f8da36cf0cc2bb69f79e0e07f4fa7988397a0b8f331063765074ed96260a6c1b WatchSource:0}: Error finding container f8da36cf0cc2bb69f79e0e07f4fa7988397a0b8f331063765074ed96260a6c1b: Status 404 returned error can't find the container with id f8da36cf0cc2bb69f79e0e07f4fa7988397a0b8f331063765074ed96260a6c1b Jul 19 14:57:59.874618 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.874593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:57:59.879577 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.879556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lhq55" Jul 19 14:57:59.881724 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.881705 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8d969b_9bc6_4405_8067_f503b71bf9d0.slice/crio-b2a26f6f0e8578f73e99c9913cc63bd7a815e7c3d0d5bce36a8ecba5cc75ef28 WatchSource:0}: Error finding container b2a26f6f0e8578f73e99c9913cc63bd7a815e7c3d0d5bce36a8ecba5cc75ef28: Status 404 returned error can't find the container with id b2a26f6f0e8578f73e99c9913cc63bd7a815e7c3d0d5bce36a8ecba5cc75ef28 Jul 19 14:57:59.885146 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.885087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bc55h" Jul 19 14:57:59.886572 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.886551 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e006bb_0ec5_41ca_a29b_ee609c919dfd.slice/crio-f7dc276f64f6dd77d26bd072284a448a7bdc5d3f665f694476bb4a46adc7cf96 WatchSource:0}: Error finding container f7dc276f64f6dd77d26bd072284a448a7bdc5d3f665f694476bb4a46adc7cf96: Status 404 returned error can't find the container with id f7dc276f64f6dd77d26bd072284a448a7bdc5d3f665f694476bb4a46adc7cf96 Jul 19 14:57:59.892164 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.892144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9g5fl" Jul 19 14:57:59.893080 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.893059 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca150b8_5079_4fb5_99b1_4133ba4ae43a.slice/crio-886dd6ae3f1462f1342d021212359e33d5292a8171cf87608fd93df7b5bf464b WatchSource:0}: Error finding container 886dd6ae3f1462f1342d021212359e33d5292a8171cf87608fd93df7b5bf464b: Status 404 returned error can't find the container with id 886dd6ae3f1462f1342d021212359e33d5292a8171cf87608fd93df7b5bf464b Jul 19 14:57:59.897665 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:57:59.897649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6qj6" Jul 19 14:57:59.899795 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.899776 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172caf8f_fd5d_4508_962d_ee98378bf734.slice/crio-38cfcfc8b70928a74d41be55f12de2f3419cc0b20cee16a369e97d6987945158 WatchSource:0}: Error finding container 38cfcfc8b70928a74d41be55f12de2f3419cc0b20cee16a369e97d6987945158: Status 404 returned error can't find the container with id 38cfcfc8b70928a74d41be55f12de2f3419cc0b20cee16a369e97d6987945158 Jul 19 14:57:59.904449 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:57:59.904426 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d7f044_9eb2_4dc3_8770_47651099e3d3.slice/crio-c1c3efcf9e2d6a532bd1a3b3b4c3d1ead138dcfc16439492a665ad56f33ce177 WatchSource:0}: Error finding container c1c3efcf9e2d6a532bd1a3b3b4c3d1ead138dcfc16439492a665ad56f33ce177: Status 404 returned error can't find the container with id c1c3efcf9e2d6a532bd1a3b3b4c3d1ead138dcfc16439492a665ad56f33ce177 Jul 19 14:58:00.095033 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.094950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:00.095205 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.095049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:00.095299 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.095179 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:00.095469 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.095344 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:01.095253834 +0000 UTC m=+2.955809805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:00.095759 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.095738 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:00.095858 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.095797 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:01.095782383 +0000 UTC m=+2.956338354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:00.196051 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.196015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:00.196225 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.196199 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:00.196225 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.196226 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:00.196345 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.196248 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:00.196345 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.196329 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:01.196310023 +0000 UTC m=+3.056865992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:00.520149 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.519973 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:52:59 +0000 UTC" deadline="2028-04-13 12:11:21.605874616 +0000 UTC" Jul 19 14:58:00.520149 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.520042 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15213h13m21.085836935s" Jul 19 14:58:00.610129 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.609649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:00.610129 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:00.609779 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:00.634813 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.634736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9g5fl" event={"ID":"172caf8f-fd5d-4508-962d-ee98378bf734","Type":"ContainerStarted","Data":"38cfcfc8b70928a74d41be55f12de2f3419cc0b20cee16a369e97d6987945158"} Jul 19 14:58:00.641611 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.641519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lhq55" event={"ID":"06e006bb-0ec5-41ca-a29b-ee609c919dfd","Type":"ContainerStarted","Data":"f7dc276f64f6dd77d26bd072284a448a7bdc5d3f665f694476bb4a46adc7cf96"} Jul 19 14:58:00.649015 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.648984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vshh6" event={"ID":"6f8d969b-9bc6-4405-8067-f503b71bf9d0","Type":"ContainerStarted","Data":"b2a26f6f0e8578f73e99c9913cc63bd7a815e7c3d0d5bce36a8ecba5cc75ef28"} Jul 19 14:58:00.659578 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.659547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x666x" event={"ID":"8717acab-00c9-46d6-b88b-fcc123b04b9c","Type":"ContainerStarted","Data":"22a13368e7607b31617fce3b21134c2f5bf35f2a8498e42b3b228c5b028f9d8d"} Jul 19 14:58:00.670356 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.670295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" event={"ID":"099f1352-0492-4608-ba23-0a6b4bb8ac0b","Type":"ContainerStarted","Data":"17a38cde8a9765ce16230c92731428a5f6ac3b28847478b22882876f4311c1d9"} Jul 19 14:58:00.680671 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.680640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6qj6" event={"ID":"53d7f044-9eb2-4dc3-8770-47651099e3d3","Type":"ContainerStarted","Data":"c1c3efcf9e2d6a532bd1a3b3b4c3d1ead138dcfc16439492a665ad56f33ce177"} Jul 19 14:58:00.695444 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.695417 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:58:00.695578 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.695509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bc55h" event={"ID":"2ca150b8-5079-4fb5-99b1-4133ba4ae43a","Type":"ContainerStarted","Data":"886dd6ae3f1462f1342d021212359e33d5292a8171cf87608fd93df7b5bf464b"} Jul 19 14:58:00.709582 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.709543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"f8da36cf0cc2bb69f79e0e07f4fa7988397a0b8f331063765074ed96260a6c1b"} Jul 19 14:58:00.731235 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:00.731194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerStarted","Data":"7e1f4d5d33ff321e1bf707056f6decbd2ffd2fbf8902dad0a917bf3a52317c33"} Jul 19 14:58:01.103986 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.103931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:01.104179 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.104024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:01.104179 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.104155 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:01.104314 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.104217 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:03.104197244 +0000 UTC m=+4.964753216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:01.104674 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.104651 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:01.104774 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.104710 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:03.104694178 +0000 UTC m=+4.965250150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:01.205314 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.205252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:01.205520 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.205450 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:01.205520 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.205474 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:01.205520 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.205486 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:01.205676 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.205546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:03.205526944 +0000 UTC m=+5.066082924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:01.521138 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.521020 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:52:59 +0000 UTC" deadline="2028-04-11 10:19:40.174809476 +0000 UTC" Jul 19 14:58:01.521138 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.521082 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15163h21m38.653731145s" Jul 19 14:58:01.607106 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.607077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:01.607313 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.607217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:01.607643 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:01.607622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:01.607740 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:01.607720 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:02.616556 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:02.616522 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:02.617126 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:02.616668 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:03.120902 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:03.120865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:03.121062 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:03.120927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:03.121062 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.121054 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:03.121183 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.121113 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:07.121095795 +0000 UTC m=+8.981651765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:03.121554 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.121531 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:03.121653 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.121591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:07.121575074 +0000 UTC m=+8.982131058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:03.222305 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:03.222192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:03.222473 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.222391 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:03.222473 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.222410 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:03.222473 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.222423 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:03.222622 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.222475 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:07.222458415 +0000 UTC m=+9.083014401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:03.607098 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:03.607009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:03.607098 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:03.607053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:03.607345 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.607172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:03.607345 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:03.607294 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:04.607516 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:04.607108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:04.607516 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:04.607245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:05.607545 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:05.607501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:05.607545 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:05.607545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:05.608054 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:05.607654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:05.608117 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:05.608091 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:06.607799 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:06.607676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:06.608298 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:06.607825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:07.158052 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:07.158011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:07.158235 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:07.158087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:07.158235 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.158214 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:07.158412 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.158216 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:07.158412 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.158299 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:15.15827262 +0000 UTC m=+17.018828600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:07.158412 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.158349 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:15.158326294 +0000 UTC m=+17.018882276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:07.259029 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:07.258986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:07.259202 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.259174 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:07.259202 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.259195 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:07.259337 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.259205 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:07.259337 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.259276 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:15.259243532 +0000 UTC m=+17.119799514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:07.607695 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:07.607612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:07.607852 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.607739 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:07.608205 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:07.608130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:07.608280 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:07.608234 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:08.608110 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:08.608073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:08.608595 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:08.608209 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:09.607552 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:09.607516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:09.607552 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:09.607550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:09.607771 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:09.607647 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:09.607843 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:09.607800 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:10.607772 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:10.607735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:10.608195 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:10.607869 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:11.607632 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:11.607600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:11.607832 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:11.607608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:11.607832 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:11.607731 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:11.607832 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:11.607818 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:12.607642 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:12.607612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:12.607830 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:12.607748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:13.607123 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:13.607087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:13.607600 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:13.607136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:13.607600 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:13.607225 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:13.607600 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:13.607353 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:14.607734 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:14.607698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:14.608086 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:14.607837 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:15.217356 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:15.217317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:15.217615 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:15.217377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:15.217615 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.217444 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:15.217615 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.217497 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:15.217615 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.217519 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:31.21749829 +0000 UTC m=+33.078054260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:15.217615 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.217549 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:31.217533547 +0000 UTC m=+33.078089517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:15.317925 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:15.317888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:15.318060 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.318044 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:15.318060 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.318060 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:15.318143 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.318070 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:15.318143 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.318124 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:31.318105892 +0000 UTC m=+33.178661877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:15.606661 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:15.606636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:15.606781 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:15.606638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:15.606781 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.606759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:15.606881 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:15.606831 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:16.606703 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:16.606670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:16.607152 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:16.606807 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:17.607656 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:17.607602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:17.608090 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:17.607735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:17.608090 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:17.607778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:17.608090 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:17.607915 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:18.608226 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:18.608196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:18.608706 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:18.608334 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:19.607543 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.607387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:19.607649 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.607387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:19.607649 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:19.607631 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:19.607723 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:19.607694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:19.791448 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.791420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" event={"ID":"a290a0755b8688c1e5c83007894063ed","Type":"ContainerStarted","Data":"4f4bd3eddc56f32a6e11cdcf185b531ef13c3f845b2c44f4859d5f5896ab53c3"} Jul 19 14:58:19.800934 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.800908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lhq55" event={"ID":"06e006bb-0ec5-41ca-a29b-ee609c919dfd","Type":"ContainerStarted","Data":"f436e4f02598eef6293ddaad1b1817dbee0c7cfb2584575aa91268cefa956278"} Jul 19 14:58:19.804042 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.804020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"5b6de4494317eecbca9b986ae041589999905cfffc984e552d5ec001ae4577c1"} Jul 19 14:58:19.804154 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.804077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"02c6b470d17ea2c88ac0f7ff59df6348fd9f7492ed091120c9344d9cd9a9464e"} Jul 19 14:58:19.804154 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.804117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"1d5f8cf91925e96e3f18cb0087d14c359f6dd233703340bf0ac08efebc1c9a3b"} Jul 19 14:58:19.804815 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.804770 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" podStartSLOduration=20.804755151 podStartE2EDuration="20.804755151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:19.804690334 +0000 UTC m=+21.665246324" watchObservedRunningTime="2026-07-19 14:58:19.804755151 +0000 UTC m=+21.665311140" Jul 19 14:58:19.819900 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:19.819861 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lhq55" podStartSLOduration=2.437568694 podStartE2EDuration="21.819847731s" totalImagesPullingTime="19.382279037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:19.819703722 +0000 UTC m=+21.680259710" watchObservedRunningTime="2026-07-19 14:58:19.819847731 +0000 UTC m=+21.680403719" Jul 19 14:58:20.609860 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.609700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:20.610033 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:20.609947 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:20.808002 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.807965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9g5fl" event={"ID":"172caf8f-fd5d-4508-962d-ee98378bf734","Type":"ContainerStarted","Data":"544460e303f12b7d7d64a0a0ff23ffee7e6b2c2ab96bc7c8398ce8334295c44d"} Jul 19 14:58:20.812845 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.812813 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vshh6" event={"ID":"6f8d969b-9bc6-4405-8067-f503b71bf9d0","Type":"ContainerStarted","Data":"936485e9277f526d60911e1a8675a2449e85b00421b13e5bba987f40204feac8"} Jul 19 14:58:20.814569 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.814544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x666x" event={"ID":"8717acab-00c9-46d6-b88b-fcc123b04b9c","Type":"ContainerStarted","Data":"2a62df5798861b525e45e953484f11b5178b0cd17b0e0d4a3a130e33f56313be"} Jul 19 14:58:20.816237 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.816211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" event={"ID":"099f1352-0492-4608-ba23-0a6b4bb8ac0b","Type":"ContainerStarted","Data":"e675909e51e0a22a466f3ec8a8d2592ede2dae8bad4821902a835652974aa872"} Jul 19 14:58:20.817813 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.817791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6qj6" event={"ID":"53d7f044-9eb2-4dc3-8770-47651099e3d3","Type":"ContainerStarted","Data":"cc2bd2cde0a1e184f4b64ec29017d2d36d631c2f51c3b2ee59d12c8a41e2a4f4"} Jul 19 14:58:20.819666 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.819638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bc55h" event={"ID":"2ca150b8-5079-4fb5-99b1-4133ba4ae43a","Type":"ContainerStarted","Data":"2ec42dc126da0c9319438f7dc6c11c9761d86f74ce04774dc3231f962596f64f"} Jul 19 14:58:20.822664 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.822643 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 14:58:20.823031 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.823012 2579 generic.go:356] "Generic (PLEG): container finished" podID="0b7adfdb-fcc8-4038-8660-cfe064a9c5a7" containerID="02c6b470d17ea2c88ac0f7ff59df6348fd9f7492ed091120c9344d9cd9a9464e" exitCode=1 Jul 19 14:58:20.823111 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.823062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerDied","Data":"02c6b470d17ea2c88ac0f7ff59df6348fd9f7492ed091120c9344d9cd9a9464e"} Jul 19 14:58:20.823166 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.823152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"080dd2b1b4eefd21b283d45f2ec4792331f0a2afead49798a4c4f99cc7c96237"} Jul 19 14:58:20.823219 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.823205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"dad5f913e4a43040a30a6b56193c31acc4eac36f088fbbd732b556938d75f67f"} Jul 19 14:58:20.823293 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.823273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"6a80df43eb87c35fdd87ecadf629d3264322304e0b56dc519cec5b70f1247bea"} Jul 19 14:58:20.824681 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.824654 2579 generic.go:356] "Generic (PLEG): container finished" podID="69db3133-e92e-47d0-9a7b-38d6501694b9" containerID="f68079600f4e94b94a989a096f168397231f9ab77c78fc18e567faae9d1c5aea" exitCode=0 Jul 19 14:58:20.824681 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.824685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerDied","Data":"f68079600f4e94b94a989a096f168397231f9ab77c78fc18e567faae9d1c5aea"} Jul 19 14:58:20.826593 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.826552 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9g5fl" podStartSLOduration=2.8981424909999998 podStartE2EDuration="22.826539901s" totalImagesPullingTime="19.92839741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:20.826314128 +0000 UTC m=+22.686870118" watchObservedRunningTime="2026-07-19 14:58:20.826539901 +0000 UTC m=+22.687095888" Jul 19 14:58:20.826674 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.826612 2579 generic.go:356] "Generic (PLEG): container finished" podID="792c18f67f33440108ee3acdda7b409b" containerID="3f762b456e36c0202422d3af892d0d27d49eda8aeb2187b880f4ac96d8ebf71d" exitCode=0 Jul 19 14:58:20.826776 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.826744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" event={"ID":"792c18f67f33440108ee3acdda7b409b","Type":"ContainerDied","Data":"3f762b456e36c0202422d3af892d0d27d49eda8aeb2187b880f4ac96d8ebf71d"} Jul 19 14:58:20.846012 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.845961 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x6qj6" podStartSLOduration=3.478353556 podStartE2EDuration="22.845941592s" totalImagesPullingTime="19.367588036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:20.845218624 +0000 UTC m=+22.705774616" watchObservedRunningTime="2026-07-19 14:58:20.845941592 +0000 UTC m=+22.706497582" Jul 19 14:58:20.859559 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.859515 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x666x" podStartSLOduration=3.428185141 podStartE2EDuration="22.859495544s" totalImagesPullingTime="19.431310403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:20.859322006 +0000 UTC m=+22.719877994" watchObservedRunningTime="2026-07-19 14:58:20.859495544 +0000 UTC m=+22.720051533" Jul 19 14:58:20.895799 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.895551 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vshh6" podStartSLOduration=3.552775537 podStartE2EDuration="22.895517033s" totalImagesPullingTime="19.342741496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:20.874717371 +0000 UTC m=+22.735273360" watchObservedRunningTime="2026-07-19 14:58:20.895517033 +0000 UTC m=+22.756073034" Jul 19 14:58:20.910750 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:20.910699 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bc55h" podStartSLOduration=3.550804683 podStartE2EDuration="22.91068549s" totalImagesPullingTime="19.359880807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:20.910545794 +0000 UTC m=+22.771101782" watchObservedRunningTime="2026-07-19 14:58:20.91068549 +0000 UTC m=+22.771241477" Jul 19 14:58:21.526717 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.526694 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 19 14:58:21.542342 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.542232 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T14:58:21.526713253Z","UUID":"2d2f09d6-0c65-4d24-b2e8-cda751221712","Handler":null,"Name":"","Endpoint":""} Jul 19 14:58:21.543761 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.543735 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 19 14:58:21.543863 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.543774 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 19 14:58:21.606765 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.606734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:21.606978 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.606734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:21.606978 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:21.606884 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:21.606978 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:21.606917 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:21.831253 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.831162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" event={"ID":"099f1352-0492-4608-ba23-0a6b4bb8ac0b","Type":"ContainerStarted","Data":"607ed9727e7875dc2c0a686137fe393b6e479d7a383bdfc29ad0b1325debec97"} Jul 19 14:58:21.833134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.833101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" event={"ID":"792c18f67f33440108ee3acdda7b409b","Type":"ContainerStarted","Data":"0e8655b4f37d29223256eac953dfa5250a0648c3b15e3bdcab68fbf26f5fcb5d"} Jul 19 14:58:21.850987 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:21.850940 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" podStartSLOduration=22.850924715 podStartE2EDuration="22.850924715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:21.850542785 +0000 UTC m=+23.711098773" watchObservedRunningTime="2026-07-19 14:58:21.850924715 +0000 UTC m=+23.711480704" Jul 19 14:58:22.606924 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:22.606883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:22.607092 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:22.607021 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:22.838370 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:22.838341 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 14:58:22.838861 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:22.838828 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"1a674ec95b66ffff20725c5dba9bf04735dfcbf38f8294efd74cf15cead7bd92"} Jul 19 14:58:22.841018 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:22.840992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" event={"ID":"099f1352-0492-4608-ba23-0a6b4bb8ac0b","Type":"ContainerStarted","Data":"89bf859a43b536d54983c54cd26bc119153f029e437f10b352f3cefb56faef1c"} Jul 19 14:58:22.861937 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:22.861842 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7xj2" podStartSLOduration=3.142012513 podStartE2EDuration="24.861822814s" totalImagesPullingTime="21.719810301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:22.861217223 +0000 UTC m=+24.721773214" watchObservedRunningTime="2026-07-19 14:58:22.861822814 +0000 UTC m=+24.722378806" Jul 19 14:58:23.607411 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:23.607382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:23.607568 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:23.607382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:23.607568 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:23.607494 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:23.607568 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:23.607550 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:23.796252 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:23.796217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:58:23.796823 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:23.796800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:58:23.843647 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:23.843613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:58:23.844169 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:23.843933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vshh6" Jul 19 14:58:24.607171 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:24.607089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:24.607353 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:24.607223 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:25.607398 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.607368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:25.607944 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:25.607473 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:25.607944 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.607526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:25.607944 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:25.607608 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:25.850845 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.850777 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 14:58:25.851221 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.851195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"0724cf85a59cd8eb8f5dce5f3e1b49bf3d0f583d730df01d32a8c1bc214c5add"} Jul 19 14:58:25.851462 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.851444 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:58:25.851664 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.851647 2579 scope.go:117] "RemoveContainer" containerID="02c6b470d17ea2c88ac0f7ff59df6348fd9f7492ed091120c9344d9cd9a9464e" Jul 19 14:58:25.853022 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.852996 2579 generic.go:356] "Generic (PLEG): container finished" podID="69db3133-e92e-47d0-9a7b-38d6501694b9" containerID="ac6dcd134aa8f486f4d32b4119e9924594412f30b43e18b721a14f5131cad505" exitCode=0 Jul 19 14:58:25.853105 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.853090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerDied","Data":"ac6dcd134aa8f486f4d32b4119e9924594412f30b43e18b721a14f5131cad505"} Jul 19 14:58:25.869446 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:25.869414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:58:26.606871 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.606842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:26.607053 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:26.606947 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:26.858322 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.858234 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 14:58:26.858817 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.858586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" event={"ID":"0b7adfdb-fcc8-4038-8660-cfe064a9c5a7","Type":"ContainerStarted","Data":"ebb918bd3bda20dd312a2304082275f235e686d615d8972dfa96e6781aca830f"} Jul 19 14:58:26.858871 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.858836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:58:26.858930 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.858915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:58:26.873452 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.873424 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:58:26.889583 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:26.889541 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" podStartSLOduration=9.445570657 podStartE2EDuration="28.889528564s" totalImagesPullingTime="19.443957907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:26.88818306 +0000 UTC m=+28.748739047" watchObservedRunningTime="2026-07-19 14:58:26.889528564 +0000 UTC m=+28.750084552" Jul 19 14:58:27.607400 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:27.607221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:27.607618 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:27.607221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:27.607618 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:27.607470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:27.607618 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:27.607533 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:27.862390 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:27.862302 2579 generic.go:356] "Generic (PLEG): container finished" podID="69db3133-e92e-47d0-9a7b-38d6501694b9" containerID="f00f786143489d932248406dc823aea00e566c3d6345022a8fd241e3a93b18b8" exitCode=0 Jul 19 14:58:27.862390 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:27.862346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerDied","Data":"f00f786143489d932248406dc823aea00e566c3d6345022a8fd241e3a93b18b8"} Jul 19 14:58:28.607332 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:28.607314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:28.607406 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:28.607392 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:28.866874 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:28.866794 2579 generic.go:356] "Generic (PLEG): container finished" podID="69db3133-e92e-47d0-9a7b-38d6501694b9" containerID="b51dc045e6d9adc810794749433aa7ab0324f76c6989098c8a3fe3f609bfe142" exitCode=0 Jul 19 14:58:28.867231 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:28.866876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerDied","Data":"b51dc045e6d9adc810794749433aa7ab0324f76c6989098c8a3fe3f609bfe142"} Jul 19 14:58:29.607502 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:29.607470 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:29.607502 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:29.607486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:29.607722 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:29.607619 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:29.607722 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:29.607631 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:30.606792 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:30.606748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:30.607219 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:30.606888 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:31.243616 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:31.243570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:31.243778 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:31.243637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:31.243778 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.243736 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:31.243899 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.243775 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:31.243899 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.243821 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs podName:b9f28b58-145b-4dcb-a3b2-82741e2ad378 nodeName:}" failed. No retries permitted until 2026-07-19 14:59:03.243798688 +0000 UTC m=+65.104354656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs") pod "network-metrics-daemon-tc8s7" (UID: "b9f28b58-145b-4dcb-a3b2-82741e2ad378") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:31.243899 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.243841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret podName:9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21 nodeName:}" failed. No retries permitted until 2026-07-19 14:59:03.243831434 +0000 UTC m=+65.104387409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret") pod "global-pull-secret-syncer-mg8z6" (UID: "9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:31.344787 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:31.344749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:31.344952 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.344931 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:31.345037 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.344956 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:31.345037 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.344969 2579 projected.go:196] Error preparing data for projected volume kube-api-access-xjskk for pod openshift-network-diagnostics/network-check-target-5jldc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:31.345128 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.345036 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk podName:c8c066bb-94e3-4586-8d1f-cb9821fcce88 nodeName:}" failed. No retries permitted until 2026-07-19 14:59:03.345016845 +0000 UTC m=+65.205572828 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjskk" (UniqueName: "kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk") pod "network-check-target-5jldc" (UID: "c8c066bb-94e3-4586-8d1f-cb9821fcce88") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:31.607496 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:31.607414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:31.607935 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.607543 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:31.607935 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:31.607549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:31.607935 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:31.607621 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:32.607649 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:32.607618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:32.608106 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:32.607739 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:33.607292 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:33.607215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:33.607403 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:33.607331 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:33.607403 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:33.607385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:33.607480 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:33.607464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:33.881515 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:33.881482 2579 generic.go:356] "Generic (PLEG): container finished" podID="69db3133-e92e-47d0-9a7b-38d6501694b9" containerID="e5a7303bfc0dcc13c2b2ef206b3b538ac1534ac299f4e41f32959aa727a26d4c" exitCode=0 Jul 19 14:58:33.881958 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:33.881533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerDied","Data":"e5a7303bfc0dcc13c2b2ef206b3b538ac1534ac299f4e41f32959aa727a26d4c"} Jul 19 14:58:34.607101 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:34.607065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:34.607290 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:34.607174 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:34.887281 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:34.887237 2579 generic.go:356] "Generic (PLEG): container finished" podID="69db3133-e92e-47d0-9a7b-38d6501694b9" containerID="dcdcc4366ff8a3cff9b009eebe2445fa7b0bd727cec1e3e47c7ac29ba6961b2e" exitCode=0 Jul 19 14:58:34.887671 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:34.887296 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerDied","Data":"dcdcc4366ff8a3cff9b009eebe2445fa7b0bd727cec1e3e47c7ac29ba6961b2e"} Jul 19 14:58:35.607430 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:35.607390 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:35.607430 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:35.607414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:35.607698 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:35.607509 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:35.607698 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:35.607621 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:35.892866 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:35.892835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" event={"ID":"69db3133-e92e-47d0-9a7b-38d6501694b9","Type":"ContainerStarted","Data":"ba9d8bb4d3341a4a371b0836b917c5f6716d611cd4e72eaccb1632367ab54ba1"} Jul 19 14:58:35.915335 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:35.915291 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8kfl7" podStartSLOduration=7.760841773 podStartE2EDuration="37.915276814s" totalImagesPullingTime="30.154435041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:35.914800816 +0000 UTC m=+37.775356804" watchObservedRunningTime="2026-07-19 14:58:35.915276814 +0000 UTC m=+37.775832797" Jul 19 14:58:36.607686 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:36.607651 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:36.607827 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:36.607762 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:37.607596 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:37.607559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:37.607960 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:37.607566 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:37.607960 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:37.607661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:37.607960 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:37.607735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:38.608023 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:38.607991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:38.608397 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:38.608074 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:39.606896 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:39.606863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:39.607040 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:39.606986 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:39.607085 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:39.607042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:39.607169 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:39.607151 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:40.606753 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:40.606717 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:40.607117 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:40.606816 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:41.606979 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:41.606946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:41.606979 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:41.606991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:41.607445 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:41.607058 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:41.607445 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:41.607166 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:42.572610 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:42.572389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5jldc"] Jul 19 14:58:42.572777 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:42.572702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:42.572836 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:42.572813 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:42.575478 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:42.575450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mg8z6"] Jul 19 14:58:42.575594 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:42.575550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:42.575692 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:42.575636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:42.576129 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:42.576098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc8s7"] Jul 19 14:58:42.576227 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:42.576191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:42.576320 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:42.576298 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:44.607145 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:44.607116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:44.607646 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:44.607120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:44.607646 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:44.607224 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc8s7" podUID="b9f28b58-145b-4dcb-a3b2-82741e2ad378" Jul 19 14:58:44.607646 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:44.607116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:44.607646 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:44.607290 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5jldc" podUID="c8c066bb-94e3-4586-8d1f-cb9821fcce88" Jul 19 14:58:44.607646 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:44.607385 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mg8z6" podUID="9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21" Jul 19 14:58:46.390378 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.390348 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeReady" Jul 19 14:58:46.390875 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.390471 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 19 14:58:46.438851 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.438817 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gpj89"] Jul 19 14:58:46.455517 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.455482 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8jltm"] Jul 19 14:58:46.455711 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.455693 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.458419 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.458394 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 19 14:58:46.458574 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.458556 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 19 14:58:46.458667 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.458618 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g45s6\"" Jul 19 14:58:46.458723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.458663 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 19 14:58:46.469560 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.469544 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gpj89"] Jul 19 14:58:46.469624 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.469569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jltm"] Jul 19 14:58:46.469697 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.469648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.472636 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.472612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 19 14:58:46.472636 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.472612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 19 14:58:46.472760 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.472644 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 19 14:58:46.472933 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.472914 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 19 14:58:46.474246 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.474216 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j74hj\"" Jul 19 14:58:46.537242 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.537182 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94kz9"] Jul 19 14:58:46.557204 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.557355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbvz\" (UniqueName: \"kubernetes.io/projected/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-kube-api-access-bbbvz\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.557355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-crio-socket\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.557355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9xzl\" (UniqueName: \"kubernetes.io/projected/f449b873-72ee-4366-97ab-65026aba1100-kube-api-access-m9xzl\") pod \"ingress-canary-gpj89\" (UID: \"f449b873-72ee-4366-97ab-65026aba1100\") " pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.557467 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.557467 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f449b873-72ee-4366-97ab-65026aba1100-cert\") pod \"ingress-canary-gpj89\" (UID: \"f449b873-72ee-4366-97ab-65026aba1100\") " pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.557467 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.557430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-data-volume\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.565554 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.565537 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94kz9"] Jul 19 14:58:46.565638 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.565628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.568316 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.568295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvk2r\"" Jul 19 14:58:46.568413 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.568345 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 19 14:58:46.568493 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.568481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 19 14:58:46.607096 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.607072 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:58:46.607231 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.607109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:58:46.607231 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.607074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:58:46.609795 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.609776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7fqw\"" Jul 19 14:58:46.609907 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.609789 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:58:46.609907 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.609822 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:58:46.610658 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.610158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:58:46.610658 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.610438 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:58:46.610658 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.610562 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hc8w7\"" Jul 19 14:58:46.638809 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.638788 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq"] Jul 19 14:58:46.658244 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbvz\" (UniqueName: \"kubernetes.io/projected/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-kube-api-access-bbbvz\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.658355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-crio-socket\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.658355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30103e94-7686-45ce-8ab5-a34c3a793cf5-tmp-dir\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.658355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30103e94-7686-45ce-8ab5-a34c3a793cf5-metrics-tls\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.658355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9xzl\" (UniqueName: \"kubernetes.io/projected/f449b873-72ee-4366-97ab-65026aba1100-kube-api-access-m9xzl\") pod \"ingress-canary-gpj89\" (UID: \"f449b873-72ee-4366-97ab-65026aba1100\") " pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.658501 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkg8q\" (UniqueName: \"kubernetes.io/projected/30103e94-7686-45ce-8ab5-a34c3a793cf5-kube-api-access-zkg8q\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.658540 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.658589 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30103e94-7686-45ce-8ab5-a34c3a793cf5-config-volume\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.658589 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-crio-socket\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.658689 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f449b873-72ee-4366-97ab-65026aba1100-cert\") pod \"ingress-canary-gpj89\" (UID: \"f449b873-72ee-4366-97ab-65026aba1100\") " pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.658738 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-data-volume\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.658738 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.658709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.659081 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.659048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-data-volume\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.659272 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.659239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.662178 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.662156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.662294 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.662240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f449b873-72ee-4366-97ab-65026aba1100-cert\") pod \"ingress-canary-gpj89\" (UID: \"f449b873-72ee-4366-97ab-65026aba1100\") " pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.663422 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.663407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq"] Jul 19 14:58:46.663492 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.663482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:46.665787 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.665766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x4k6p\"" Jul 19 14:58:46.665878 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.665765 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 19 14:58:46.665878 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.665772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9xzl\" (UniqueName: \"kubernetes.io/projected/f449b873-72ee-4366-97ab-65026aba1100-kube-api-access-m9xzl\") pod \"ingress-canary-gpj89\" (UID: \"f449b873-72ee-4366-97ab-65026aba1100\") " pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.665989 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.665898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbvz\" (UniqueName: \"kubernetes.io/projected/e86b30e8-a3f4-425b-84e7-8fa8f7bf6531-kube-api-access-bbbvz\") pod \"insights-runtime-extractor-8jltm\" (UID: \"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531\") " pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.759626 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.759597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkg8q\" (UniqueName: \"kubernetes.io/projected/30103e94-7686-45ce-8ab5-a34c3a793cf5-kube-api-access-zkg8q\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.759763 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.759642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30103e94-7686-45ce-8ab5-a34c3a793cf5-config-volume\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.759763 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.759664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4de4a976-1d92-4780-8c1b-f6e976d307ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-ttjqq\" (UID: \"4de4a976-1d92-4780-8c1b-f6e976d307ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:46.759763 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.759747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30103e94-7686-45ce-8ab5-a34c3a793cf5-tmp-dir\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.759859 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.759780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30103e94-7686-45ce-8ab5-a34c3a793cf5-metrics-tls\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.760083 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.760062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30103e94-7686-45ce-8ab5-a34c3a793cf5-tmp-dir\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.760170 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.760145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30103e94-7686-45ce-8ab5-a34c3a793cf5-config-volume\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.761910 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.761891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30103e94-7686-45ce-8ab5-a34c3a793cf5-metrics-tls\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.764847 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.764832 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gpj89" Jul 19 14:58:46.767298 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.767278 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkg8q\" (UniqueName: \"kubernetes.io/projected/30103e94-7686-45ce-8ab5-a34c3a793cf5-kube-api-access-zkg8q\") pod \"dns-default-94kz9\" (UID: \"30103e94-7686-45ce-8ab5-a34c3a793cf5\") " pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.777231 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.777208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jltm" Jul 19 14:58:46.860232 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.860193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4de4a976-1d92-4780-8c1b-f6e976d307ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-ttjqq\" (UID: \"4de4a976-1d92-4780-8c1b-f6e976d307ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:46.862573 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.862542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4de4a976-1d92-4780-8c1b-f6e976d307ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-ttjqq\" (UID: \"4de4a976-1d92-4780-8c1b-f6e976d307ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:46.874779 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.874385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:46.979167 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.979049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:46.979882 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.979862 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gpj89"] Jul 19 14:58:46.983182 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:46.983153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jltm"] Jul 19 14:58:46.989529 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:58:46.989442 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86b30e8_a3f4_425b_84e7_8fa8f7bf6531.slice/crio-ba8652ac785bfcb10dc971c523e2d1994719210870ff9c665397c12cb45b428e WatchSource:0}: Error finding container ba8652ac785bfcb10dc971c523e2d1994719210870ff9c665397c12cb45b428e: Status 404 returned error can't find the container with id ba8652ac785bfcb10dc971c523e2d1994719210870ff9c665397c12cb45b428e Jul 19 14:58:47.038598 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:47.038443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94kz9"] Jul 19 14:58:47.042738 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:58:47.042707 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30103e94_7686_45ce_8ab5_a34c3a793cf5.slice/crio-964a6c9594f089d5b13ef5556f426ad34b1f8c5657c732493aaf21086f7fec69 WatchSource:0}: Error finding container 964a6c9594f089d5b13ef5556f426ad34b1f8c5657c732493aaf21086f7fec69: Status 404 returned error can't find the container with id 964a6c9594f089d5b13ef5556f426ad34b1f8c5657c732493aaf21086f7fec69 Jul 19 14:58:47.112720 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:47.112692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq"] Jul 19 14:58:47.115892 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:58:47.115864 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de4a976_1d92_4780_8c1b_f6e976d307ee.slice/crio-a14fdf7cf172cac702311b77540bc90cb68d694ac9c66f43bedadfab4878e0ec WatchSource:0}: Error finding container a14fdf7cf172cac702311b77540bc90cb68d694ac9c66f43bedadfab4878e0ec: Status 404 returned error can't find the container with id a14fdf7cf172cac702311b77540bc90cb68d694ac9c66f43bedadfab4878e0ec Jul 19 14:58:47.926045 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:47.925943 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gpj89" event={"ID":"f449b873-72ee-4366-97ab-65026aba1100","Type":"ContainerStarted","Data":"628de69fa333fe44cb6042c522f453d3549100a2481716d98cf2fdbc052ce75b"} Jul 19 14:58:47.929511 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:47.929475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" event={"ID":"4de4a976-1d92-4780-8c1b-f6e976d307ee","Type":"ContainerStarted","Data":"a14fdf7cf172cac702311b77540bc90cb68d694ac9c66f43bedadfab4878e0ec"} Jul 19 14:58:47.932392 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:47.932363 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94kz9" event={"ID":"30103e94-7686-45ce-8ab5-a34c3a793cf5","Type":"ContainerStarted","Data":"964a6c9594f089d5b13ef5556f426ad34b1f8c5657c732493aaf21086f7fec69"} Jul 19 14:58:47.935176 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:47.935151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jltm" event={"ID":"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531","Type":"ContainerStarted","Data":"ba8652ac785bfcb10dc971c523e2d1994719210870ff9c665397c12cb45b428e"} Jul 19 14:58:48.776481 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.776408 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff48696c7-9b7m2"] Jul 19 14:58:48.792109 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.792077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.794581 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.794548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff48696c7-9b7m2"] Jul 19 14:58:48.796224 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.796198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 19 14:58:48.796381 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.796361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 19 14:58:48.796644 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.796626 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bffv5\"" Jul 19 14:58:48.797023 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.797001 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 19 14:58:48.797124 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.797056 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 19 14:58:48.798841 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.797244 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 19 14:58:48.798841 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.797587 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 19 14:58:48.798841 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.797846 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 19 14:58:48.878561 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.878517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-config\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.878733 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.878583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-serving-cert\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.878733 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.878620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9k49\" (UniqueName: \"kubernetes.io/projected/2e41acde-c4b9-43da-874f-3a46bf6d1b43-kube-api-access-s9k49\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.878733 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.878694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-service-ca\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.878733 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.878727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-oauth-serving-cert\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.878998 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.878765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-oauth-config\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.979625 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.979589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-oauth-config\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980071 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.979701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-config\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980071 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.979738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-serving-cert\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980071 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.979768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9k49\" (UniqueName: \"kubernetes.io/projected/2e41acde-c4b9-43da-874f-3a46bf6d1b43-kube-api-access-s9k49\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980071 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.979807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-service-ca\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980071 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.979823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-oauth-serving-cert\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980585 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.980555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-oauth-serving-cert\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980826 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.980787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-service-ca\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.980939 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.980867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-config\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.984630 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.984445 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-oauth-config\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.984630 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.984449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-serving-cert\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:48.989054 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:48.989028 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9k49\" (UniqueName: \"kubernetes.io/projected/2e41acde-c4b9-43da-874f-3a46bf6d1b43-kube-api-access-s9k49\") pod \"console-ff48696c7-9b7m2\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:49.112901 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:49.112800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:53.071204 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.071132 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff48696c7-9b7m2"] Jul 19 14:58:53.075109 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:58:53.075078 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e41acde_c4b9_43da_874f_3a46bf6d1b43.slice/crio-c11de5843f62d7272dd4de3e25d0794de9a9a28db87187fc1047469de686ade7 WatchSource:0}: Error finding container c11de5843f62d7272dd4de3e25d0794de9a9a28db87187fc1047469de686ade7: Status 404 returned error can't find the container with id c11de5843f62d7272dd4de3e25d0794de9a9a28db87187fc1047469de686ade7 Jul 19 14:58:53.956751 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.956233 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" event={"ID":"4de4a976-1d92-4780-8c1b-f6e976d307ee","Type":"ContainerStarted","Data":"4c26b4d315af2a2a9575c40303f1d1e4b478b566d494df28b0e395dfcd40c84a"} Jul 19 14:58:53.956751 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.956317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:53.962329 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.961692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94kz9" event={"ID":"30103e94-7686-45ce-8ab5-a34c3a793cf5","Type":"ContainerStarted","Data":"2be334a17cd0d21e33b06550b5e4ec5ff7eda628ac161f5385f82537ed3031db"} Jul 19 14:58:53.962329 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.961744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94kz9" event={"ID":"30103e94-7686-45ce-8ab5-a34c3a793cf5","Type":"ContainerStarted","Data":"4aad4c92be5c22127655fc0768f499f2ddb587d68b0756ad846b833bdd57552d"} Jul 19 14:58:53.962329 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.962214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94kz9" Jul 19 14:58:53.964097 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.964075 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" Jul 19 14:58:53.966022 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.965996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jltm" event={"ID":"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531","Type":"ContainerStarted","Data":"661e61c4b4000e7d04c43786722362e82872facfe3d14de01f08d6e1a84140b7"} Jul 19 14:58:53.968168 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.968141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gpj89" event={"ID":"f449b873-72ee-4366-97ab-65026aba1100","Type":"ContainerStarted","Data":"14fd8f2f50fce3a69881f329048b561038005fc415ee9b3618ac00ab9fff98f9"} Jul 19 14:58:53.969961 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.969933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff48696c7-9b7m2" event={"ID":"2e41acde-c4b9-43da-874f-3a46bf6d1b43","Type":"ContainerStarted","Data":"c11de5843f62d7272dd4de3e25d0794de9a9a28db87187fc1047469de686ade7"} Jul 19 14:58:53.972880 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.972835 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ttjqq" podStartSLOduration=2.838508451 podStartE2EDuration="7.972818611s" totalImagesPullingTime="5.13431016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:53.972281879 +0000 UTC m=+55.832837865" watchObservedRunningTime="2026-07-19 14:58:53.972818611 +0000 UTC m=+55.833374595" Jul 19 14:58:53.988234 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:53.988187 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gpj89" podStartSLOduration=2.105907851 podStartE2EDuration="7.988174861s" totalImagesPullingTime="5.88226701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:53.988019497 +0000 UTC m=+55.848575486" watchObservedRunningTime="2026-07-19 14:58:53.988174861 +0000 UTC m=+55.848730827" Jul 19 14:58:54.023774 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.023592 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-94kz9" podStartSLOduration=2.199700131 podStartE2EDuration="8.023572472s" totalImagesPullingTime="5.823872341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:54.022438871 +0000 UTC m=+55.882994860" watchObservedRunningTime="2026-07-19 14:58:54.023572472 +0000 UTC m=+55.884128465" Jul 19 14:58:54.653290 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.653166 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-br69q"] Jul 19 14:58:54.664779 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.664746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.667418 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.667350 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-br69q"] Jul 19 14:58:54.668677 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.668652 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 19 14:58:54.668852 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.668660 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w8nqh\"" Jul 19 14:58:54.669983 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.669914 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 19 14:58:54.670125 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.670021 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 19 14:58:54.670214 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.670138 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 19 14:58:54.670787 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.670764 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 19 14:58:54.727983 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.727943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.728171 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.728085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.728171 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.728138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caf614ce-4614-4ac4-9884-3c0a32f512c0-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.728171 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.728162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4jm\" (UniqueName: \"kubernetes.io/projected/caf614ce-4614-4ac4-9884-3c0a32f512c0-kube-api-access-qc4jm\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.829389 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.829357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.829567 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.829408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caf614ce-4614-4ac4-9884-3c0a32f512c0-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.829567 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.829435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4jm\" (UniqueName: \"kubernetes.io/projected/caf614ce-4614-4ac4-9884-3c0a32f512c0-kube-api-access-qc4jm\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.829567 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.829503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.829567 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:54.829555 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 19 14:58:54.829790 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:58:54.829631 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-tls podName:caf614ce-4614-4ac4-9884-3c0a32f512c0 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:55.329610214 +0000 UTC m=+57.190166203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-br69q" (UID: "caf614ce-4614-4ac4-9884-3c0a32f512c0") : secret "prometheus-operator-tls" not found Jul 19 14:58:54.830617 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.830587 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caf614ce-4614-4ac4-9884-3c0a32f512c0-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.832966 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.832940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:54.845603 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:54.845579 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4jm\" (UniqueName: \"kubernetes.io/projected/caf614ce-4614-4ac4-9884-3c0a32f512c0-kube-api-access-qc4jm\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:55.334475 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:55.334432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:55.341715 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:55.337354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/caf614ce-4614-4ac4-9884-3c0a32f512c0-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-br69q\" (UID: \"caf614ce-4614-4ac4-9884-3c0a32f512c0\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:55.578107 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:55.578065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" Jul 19 14:58:57.363336 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.363299 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc5dd7fc6-ksgfb"] Jul 19 14:58:57.370630 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.370602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.379029 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.378927 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc5dd7fc6-ksgfb"] Jul 19 14:58:57.381476 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.381443 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 19 14:58:57.454522 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6h8\" (UniqueName: \"kubernetes.io/projected/41aa65a5-39ba-489c-9108-bfd928bf5f17-kube-api-access-ms6h8\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.454718 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-service-ca\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.454718 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-oauth-serving-cert\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.454718 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-trusted-ca-bundle\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.454718 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-oauth-config\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.454718 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-serving-cert\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.454982 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.454715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-config\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556070 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-oauth-config\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556070 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-serving-cert\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556272 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-config\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556272 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6h8\" (UniqueName: \"kubernetes.io/projected/41aa65a5-39ba-489c-9108-bfd928bf5f17-kube-api-access-ms6h8\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556272 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-service-ca\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556272 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-oauth-serving-cert\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556469 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-trusted-ca-bundle\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.556964 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.556932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-config\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.557114 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.557095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-service-ca\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.557182 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.557095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-oauth-serving-cert\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.557275 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.557237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-trusted-ca-bundle\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.558532 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.558511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-serving-cert\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.558659 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.558582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-oauth-config\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.564048 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.564025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6h8\" (UniqueName: \"kubernetes.io/projected/41aa65a5-39ba-489c-9108-bfd928bf5f17-kube-api-access-ms6h8\") pod \"console-cc5dd7fc6-ksgfb\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.682755 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.682720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:58:57.811981 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.811943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-br69q"] Jul 19 14:58:57.816288 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:58:57.816217 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf614ce_4614_4ac4_9884_3c0a32f512c0.slice/crio-1e1b99c2dc953f3bb2c6e826cf261ccd9a4cb359818f198269dcf6bb0ab6eeee WatchSource:0}: Error finding container 1e1b99c2dc953f3bb2c6e826cf261ccd9a4cb359818f198269dcf6bb0ab6eeee: Status 404 returned error can't find the container with id 1e1b99c2dc953f3bb2c6e826cf261ccd9a4cb359818f198269dcf6bb0ab6eeee Jul 19 14:58:57.832642 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.832612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc5dd7fc6-ksgfb"] Jul 19 14:58:57.836141 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:58:57.836113 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41aa65a5_39ba_489c_9108_bfd928bf5f17.slice/crio-ea0162bbfdcd7512d79319f1cb97f9751fb3a904e7e823127cc2dd292070a497 WatchSource:0}: Error finding container ea0162bbfdcd7512d79319f1cb97f9751fb3a904e7e823127cc2dd292070a497: Status 404 returned error can't find the container with id ea0162bbfdcd7512d79319f1cb97f9751fb3a904e7e823127cc2dd292070a497 Jul 19 14:58:57.985367 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.985285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" event={"ID":"caf614ce-4614-4ac4-9884-3c0a32f512c0","Type":"ContainerStarted","Data":"1e1b99c2dc953f3bb2c6e826cf261ccd9a4cb359818f198269dcf6bb0ab6eeee"} Jul 19 14:58:57.986887 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.986850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5dd7fc6-ksgfb" event={"ID":"41aa65a5-39ba-489c-9108-bfd928bf5f17","Type":"ContainerStarted","Data":"ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0"} Jul 19 14:58:57.987018 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.986914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5dd7fc6-ksgfb" event={"ID":"41aa65a5-39ba-489c-9108-bfd928bf5f17","Type":"ContainerStarted","Data":"ea0162bbfdcd7512d79319f1cb97f9751fb3a904e7e823127cc2dd292070a497"} Jul 19 14:58:57.988575 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.988532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jltm" event={"ID":"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531","Type":"ContainerStarted","Data":"bf82891d091b8b5bc6a68929c7b40299a12d86febef0f8541913a88aa0f97a92"} Jul 19 14:58:57.989877 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:57.989857 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff48696c7-9b7m2" event={"ID":"2e41acde-c4b9-43da-874f-3a46bf6d1b43","Type":"ContainerStarted","Data":"20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b"} Jul 19 14:58:58.008599 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:58.008545 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cc5dd7fc6-ksgfb" podStartSLOduration=1.008528277 podStartE2EDuration="1.008528277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:58.00803403 +0000 UTC m=+59.868590019" watchObservedRunningTime="2026-07-19 14:58:58.008528277 +0000 UTC m=+59.869084263" Jul 19 14:58:58.024170 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:58.024129 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff48696c7-9b7m2" podStartSLOduration=5.412512688 podStartE2EDuration="10.024116258s" totalImagesPullingTime="4.61160357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:58.023494657 +0000 UTC m=+59.884050648" watchObservedRunningTime="2026-07-19 14:58:58.024116258 +0000 UTC m=+59.884672245" Jul 19 14:58:58.879415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:58.879334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzhcm" Jul 19 14:58:59.113181 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:59.113143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:59.113384 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:59.113224 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:59.119560 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:59.119529 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:58:59.999102 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:59.999059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" event={"ID":"caf614ce-4614-4ac4-9884-3c0a32f512c0","Type":"ContainerStarted","Data":"f48b6ed547c47557c3feb2870387d9b2c35c6a61c429b02abf86ccb125833819"} Jul 19 14:58:59.999576 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:58:59.999117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" event={"ID":"caf614ce-4614-4ac4-9884-3c0a32f512c0","Type":"ContainerStarted","Data":"d627e5e5d6623cd127499c6f84b3ae9e8667cacc2bdf819bd9b57ad644cdce40"} Jul 19 14:59:00.004496 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:00.004472 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:59:00.020384 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:00.020324 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-br69q" podStartSLOduration=4.680884088 podStartE2EDuration="6.02030322s" totalImagesPullingTime="1.339419132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:00.018830723 +0000 UTC m=+61.879386714" watchObservedRunningTime="2026-07-19 14:59:00.02030322 +0000 UTC m=+61.880859210" Jul 19 14:59:01.971142 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.971104 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4"] Jul 19 14:59:01.975092 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.975067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:01.978124 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.978094 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 19 14:59:01.978245 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.978133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xkfg8\"" Jul 19 14:59:01.978245 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.978220 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 19 14:59:01.985024 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.984994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4"] Jul 19 14:59:01.998176 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:01.998138 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cd8nd"] Jul 19 14:59:02.002092 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.002068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.004361 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.004335 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-n9kmq"] Jul 19 14:59:02.005026 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.004994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 19 14:59:02.005295 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.005243 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 19 14:59:02.005459 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.005437 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 19 14:59:02.005825 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.005806 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tcft9\"" Jul 19 14:59:02.008899 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.008864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jltm" event={"ID":"e86b30e8-a3f4-425b-84e7-8fa8f7bf6531","Type":"ContainerStarted","Data":"aa011056514829ca6c6065ce61115ee0b965c2e1add7931baacfd896d2366549"} Jul 19 14:59:02.009015 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.008974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.011354 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.011336 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-n6vrc\"" Jul 19 14:59:02.011642 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.011617 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 19 14:59:02.011774 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.011757 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 19 14:59:02.011849 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.011833 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 19 14:59:02.020103 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.020079 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-n9kmq"] Jul 19 14:59:02.045439 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.045388 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8jltm" podStartSLOduration=1.898889335 podStartE2EDuration="16.045372333s" totalImagesPullingTime="14.146482998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:58:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:02.044880247 +0000 UTC m=+63.905436236" watchObservedRunningTime="2026-07-19 14:59:02.045372333 +0000 UTC m=+63.905928349" Jul 19 14:59:02.096946 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.096908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvxj\" (UniqueName: \"kubernetes.io/projected/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-api-access-6bvxj\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.097102 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.096989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097102 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcp8\" (UniqueName: \"kubernetes.io/projected/b5b7f146-7844-4978-b6ea-e2e61559fe5a-kube-api-access-4bcp8\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097102 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-wtmp\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097305 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-sys\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097353 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.097474 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-textfile\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097519 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.097572 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097626 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.097668 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-tls\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097707 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6a2f39-a519-474e-889c-28bbfd94bf4e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.097754 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.097793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc6a2f39-a519-474e-889c-28bbfd94bf4e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.097793 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7367334-c047-4877-a867-b9fd05c218ac-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.097878 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-root\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.097878 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/a7367334-c047-4877-a867-b9fd05c218ac-kube-api-access-qrgcc\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.097974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.097974 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.097899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7f146-7844-4978-b6ea-e2e61559fe5a-metrics-client-ca\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.199276 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-textfile\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.199450 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.199450 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.199450 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.199609 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-tls\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.199609 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6a2f39-a519-474e-889c-28bbfd94bf4e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.199709 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:02.199608 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 19 14:59:02.199709 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.199709 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:02.199669 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-tls podName:b5b7f146-7844-4978-b6ea-e2e61559fe5a nodeName:}" failed. No retries permitted until 2026-07-19 14:59:02.699653368 +0000 UTC m=+64.560209334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-tls") pod "node-exporter-cd8nd" (UID: "b5b7f146-7844-4978-b6ea-e2e61559fe5a") : secret "node-exporter-tls" not found Jul 19 14:59:02.199709 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-textfile\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.199709 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc6a2f39-a519-474e-889c-28bbfd94bf4e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:02.199732 2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7367334-c047-4877-a867-b9fd05c218ac-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:02.199784 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-tls podName:a7367334-c047-4877-a867-b9fd05c218ac nodeName:}" failed. No retries permitted until 2026-07-19 14:59:02.699766965 +0000 UTC m=+64.560322949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-sl8v4" (UID: "a7367334-c047-4877-a867-b9fd05c218ac") : secret "openshift-state-metrics-tls" not found Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-root\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/a7367334-c047-4877-a867-b9fd05c218ac-kube-api-access-qrgcc\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.199962 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7f146-7844-4978-b6ea-e2e61559fe5a-metrics-client-ca\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvxj\" (UniqueName: \"kubernetes.io/projected/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-api-access-6bvxj\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199983 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fc6a2f39-a519-474e-889c-28bbfd94bf4e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.199997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcp8\" (UniqueName: \"kubernetes.io/projected/b5b7f146-7844-4978-b6ea-e2e61559fe5a-kube-api-access-4bcp8\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-wtmp\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:02.200094 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6a2f39-a519-474e-889c-28bbfd94bf4e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-sys\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-root\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:02.200139 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-tls podName:fc6a2f39-a519-474e-889c-28bbfd94bf4e nodeName:}" failed. No retries permitted until 2026-07-19 14:59:02.700125626 +0000 UTC m=+64.560681597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-n9kmq" (UID: "fc6a2f39-a519-474e-889c-28bbfd94bf4e") : secret "kube-state-metrics-tls" not found Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-sys\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.200372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-wtmp\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.201091 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7367334-c047-4877-a867-b9fd05c218ac-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.201091 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.200742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b7f146-7844-4978-b6ea-e2e61559fe5a-metrics-client-ca\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.202194 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.202162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.202339 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.202210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.202339 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.202326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.210823 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.210794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgcc\" (UniqueName: \"kubernetes.io/projected/a7367334-c047-4877-a867-b9fd05c218ac-kube-api-access-qrgcc\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.211355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.211329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcp8\" (UniqueName: \"kubernetes.io/projected/b5b7f146-7844-4978-b6ea-e2e61559fe5a-kube-api-access-4bcp8\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.211555 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.211538 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvxj\" (UniqueName: \"kubernetes.io/projected/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-api-access-6bvxj\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.704193 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.704152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.704419 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.704239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-tls\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.704419 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.704278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.706781 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.706752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b7f146-7844-4978-b6ea-e2e61559fe5a-node-exporter-tls\") pod \"node-exporter-cd8nd\" (UID: \"b5b7f146-7844-4978-b6ea-e2e61559fe5a\") " pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.706919 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.706804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7367334-c047-4877-a867-b9fd05c218ac-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sl8v4\" (UID: \"a7367334-c047-4877-a867-b9fd05c218ac\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.706919 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.706817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc6a2f39-a519-474e-889c-28bbfd94bf4e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-n9kmq\" (UID: \"fc6a2f39-a519-474e-889c-28bbfd94bf4e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.884225 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.884190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" Jul 19 14:59:02.912536 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.912505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cd8nd" Jul 19 14:59:02.919101 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:02.918860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" Jul 19 14:59:02.947642 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:02.947613 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b7f146_7844_4978_b6ea_e2e61559fe5a.slice/crio-b59c0c277600a1977bc2a600ff1c00e43bc6a41722c26ead8ec84871d3ff512d WatchSource:0}: Error finding container b59c0c277600a1977bc2a600ff1c00e43bc6a41722c26ead8ec84871d3ff512d: Status 404 returned error can't find the container with id b59c0c277600a1977bc2a600ff1c00e43bc6a41722c26ead8ec84871d3ff512d Jul 19 14:59:03.015663 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.015063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cd8nd" event={"ID":"b5b7f146-7844-4978-b6ea-e2e61559fe5a","Type":"ContainerStarted","Data":"b59c0c277600a1977bc2a600ff1c00e43bc6a41722c26ead8ec84871d3ff512d"} Jul 19 14:59:03.042761 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.042729 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:59:03.077053 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.077023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4"] Jul 19 14:59:03.077161 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.077069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:59:03.077300 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.077271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.080078 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 19 14:59:03.080188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 19 14:59:03.080188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080068 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 19 14:59:03.080379 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080363 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 19 14:59:03.080445 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080409 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 19 14:59:03.080490 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080454 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 19 14:59:03.080775 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080755 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 19 14:59:03.080863 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080805 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 19 14:59:03.080923 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.080862 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 19 14:59:03.081103 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.081084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2gnrw\"" Jul 19 14:59:03.119320 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.119293 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-n9kmq"] Jul 19 14:59:03.122631 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:03.122601 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6a2f39_a519_474e_889c_28bbfd94bf4e.slice/crio-18a37f64c20a367d6feb033758beb5eabc66182d08e466f10bf04cfced7bcbb3 WatchSource:0}: Error finding container 18a37f64c20a367d6feb033758beb5eabc66182d08e466f10bf04cfced7bcbb3: Status 404 returned error can't find the container with id 18a37f64c20a367d6feb033758beb5eabc66182d08e466f10bf04cfced7bcbb3 Jul 19 14:59:03.207377 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207476 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207476 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207476 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207623 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207623 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207623 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psj6l\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-kube-api-access-psj6l\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207722 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207722 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207722 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207816 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207816 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.207874 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.207817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308577 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308701 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308861 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308861 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308861 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psj6l\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-kube-api-access-psj6l\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.308861 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309044 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309044 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309044 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.308967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309044 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309218 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309218 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:59:03.309218 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:59:03.309422 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309650 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.309730 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.309682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.311713 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.311645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.311713 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.311647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.311893 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.311864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.312017 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.311921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.312355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.312318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.312560 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.312526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-out\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.312831 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.312812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.312923 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.312857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-volume\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.312975 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.312931 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:59:03.313047 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.312933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:59:03.313633 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.313616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-web-config\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.318326 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.318309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psj6l\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-kube-api-access-psj6l\") pod \"alertmanager-main-0\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.321724 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.321700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21-original-pull-secret\") pod \"global-pull-secret-syncer-mg8z6\" (UID: \"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21\") " pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:59:03.321862 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.321846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9f28b58-145b-4dcb-a3b2-82741e2ad378-metrics-certs\") pod \"network-metrics-daemon-tc8s7\" (UID: \"b9f28b58-145b-4dcb-a3b2-82741e2ad378\") " pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:59:03.389807 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.389774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 14:59:03.410147 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.410114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:59:03.412792 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.412770 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:59:03.419024 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.419009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mg8z6" Jul 19 14:59:03.423289 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.423251 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:59:03.431313 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.431288 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hc8w7\"" Jul 19 14:59:03.433508 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.433486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjskk\" (UniqueName: \"kubernetes.io/projected/c8c066bb-94e3-4586-8d1f-cb9821fcce88-kube-api-access-xjskk\") pod \"network-check-target-5jldc\" (UID: \"c8c066bb-94e3-4586-8d1f-cb9821fcce88\") " pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:59:03.439375 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.439358 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc8s7" Jul 19 14:59:03.575568 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.575466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 14:59:03.577703 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:03.577671 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2ef8aa_4065_4d79_8c6e_27f2c3e2d25f.slice/crio-8e0eb799b07701eac4829f2984e44d1c191a02f40e3c68136b83345bbee668ab WatchSource:0}: Error finding container 8e0eb799b07701eac4829f2984e44d1c191a02f40e3c68136b83345bbee668ab: Status 404 returned error can't find the container with id 8e0eb799b07701eac4829f2984e44d1c191a02f40e3c68136b83345bbee668ab Jul 19 14:59:03.602704 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.602659 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mg8z6"] Jul 19 14:59:03.604819 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:03.604791 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cadb2b0_9a6e_4d64_8d1e_8b5bb0aefe21.slice/crio-ebcfc9da51ecd3bc9ed3da2883dd7d18d9a5bab0ea50eb15f92be09fcc81ebfe WatchSource:0}: Error finding container ebcfc9da51ecd3bc9ed3da2883dd7d18d9a5bab0ea50eb15f92be09fcc81ebfe: Status 404 returned error can't find the container with id ebcfc9da51ecd3bc9ed3da2883dd7d18d9a5bab0ea50eb15f92be09fcc81ebfe Jul 19 14:59:03.629888 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.629846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc8s7"] Jul 19 14:59:03.633688 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:03.633663 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f28b58_145b_4dcb_a3b2_82741e2ad378.slice/crio-b8e6c8bb77acb6d3bc108cede5a66e6d86276b3b509b59de935ab50426755374 WatchSource:0}: Error finding container b8e6c8bb77acb6d3bc108cede5a66e6d86276b3b509b59de935ab50426755374: Status 404 returned error can't find the container with id b8e6c8bb77acb6d3bc108cede5a66e6d86276b3b509b59de935ab50426755374 Jul 19 14:59:03.727164 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.727084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7fqw\"" Jul 19 14:59:03.735586 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.735334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:59:03.886145 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:03.885979 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5jldc"] Jul 19 14:59:04.002394 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:04.002318 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c066bb_94e3_4586_8d1f_cb9821fcce88.slice/crio-16bc298dddbb58f8ba06ca9845458eb176951307c227fe0499db7f430520515c WatchSource:0}: Error finding container 16bc298dddbb58f8ba06ca9845458eb176951307c227fe0499db7f430520515c: Status 404 returned error can't find the container with id 16bc298dddbb58f8ba06ca9845458eb176951307c227fe0499db7f430520515c Jul 19 14:59:04.020922 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.020887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"8e0eb799b07701eac4829f2984e44d1c191a02f40e3c68136b83345bbee668ab"} Jul 19 14:59:04.024071 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.024039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" event={"ID":"fc6a2f39-a519-474e-889c-28bbfd94bf4e","Type":"ContainerStarted","Data":"18a37f64c20a367d6feb033758beb5eabc66182d08e466f10bf04cfced7bcbb3"} Jul 19 14:59:04.026318 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.026252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc8s7" event={"ID":"b9f28b58-145b-4dcb-a3b2-82741e2ad378","Type":"ContainerStarted","Data":"b8e6c8bb77acb6d3bc108cede5a66e6d86276b3b509b59de935ab50426755374"} Jul 19 14:59:04.028089 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.028061 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mg8z6" event={"ID":"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21","Type":"ContainerStarted","Data":"ebcfc9da51ecd3bc9ed3da2883dd7d18d9a5bab0ea50eb15f92be09fcc81ebfe"} Jul 19 14:59:04.029563 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.029528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5jldc" event={"ID":"c8c066bb-94e3-4586-8d1f-cb9821fcce88","Type":"ContainerStarted","Data":"16bc298dddbb58f8ba06ca9845458eb176951307c227fe0499db7f430520515c"} Jul 19 14:59:04.031366 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.031346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" event={"ID":"a7367334-c047-4877-a867-b9fd05c218ac","Type":"ContainerStarted","Data":"1d2b70525351269f271ee71f1a8fdda74e4f9b8de7e52a2e47b74fecff034494"} Jul 19 14:59:04.031441 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.031386 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" event={"ID":"a7367334-c047-4877-a867-b9fd05c218ac","Type":"ContainerStarted","Data":"f0e569417c9d25ac4e7cd83cc5fad096a8633c1bdb78b96303c35c6968548597"} Jul 19 14:59:04.031441 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.031411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" event={"ID":"a7367334-c047-4877-a867-b9fd05c218ac","Type":"ContainerStarted","Data":"d816606168d3a58fa3de6236f574032034df69305f1fff1e1515660c3e7d3aff"} Jul 19 14:59:04.969148 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.968291 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78db44bb49-2slcq"] Jul 19 14:59:04.971788 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.971370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:04.974008 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.973918 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 19 14:59:04.974155 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.974070 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 19 14:59:04.974222 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.974163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 19 14:59:04.974222 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.974171 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 19 14:59:04.974222 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.974182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 19 14:59:04.975662 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.975639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x6jrx\"" Jul 19 14:59:04.985621 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.982407 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2ulo6nfnst0pg\"" Jul 19 14:59:04.985621 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.984738 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78db44bb49-2slcq"] Jul 19 14:59:04.987371 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:04.986804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94kz9" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlqf\" (UniqueName: \"kubernetes.io/projected/0c17fba3-b20b-4128-b203-d96fa6cb5aea-kube-api-access-qwlqf\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c17fba3-b20b-4128-b203-d96fa6cb5aea-metrics-client-ca\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-tls\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-grpc-tls\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.024932 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.024835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.039817 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.039780 2579 generic.go:356] "Generic (PLEG): container finished" podID="b5b7f146-7844-4978-b6ea-e2e61559fe5a" containerID="32b95584ba85174f46d89e2a89c1e0922122a0e671b212a55fa82cef97387e55" exitCode=0 Jul 19 14:59:05.039980 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.039872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cd8nd" event={"ID":"b5b7f146-7844-4978-b6ea-e2e61559fe5a","Type":"ContainerDied","Data":"32b95584ba85174f46d89e2a89c1e0922122a0e671b212a55fa82cef97387e55"} Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlqf\" (UniqueName: \"kubernetes.io/projected/0c17fba3-b20b-4128-b203-d96fa6cb5aea-kube-api-access-qwlqf\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c17fba3-b20b-4128-b203-d96fa6cb5aea-metrics-client-ca\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-tls\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-grpc-tls\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.127842 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.126693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.128382 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.128222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c17fba3-b20b-4128-b203-d96fa6cb5aea-metrics-client-ca\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.133951 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.132999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.133951 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.133079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-grpc-tls\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.133951 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.133671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.133951 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.133670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.134238 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.134078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-tls\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.134777 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.134724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c17fba3-b20b-4128-b203-d96fa6cb5aea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.139734 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.139675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlqf\" (UniqueName: \"kubernetes.io/projected/0c17fba3-b20b-4128-b203-d96fa6cb5aea-kube-api-access-qwlqf\") pod \"thanos-querier-78db44bb49-2slcq\" (UID: \"0c17fba3-b20b-4128-b203-d96fa6cb5aea\") " pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:05.288888 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:05.288797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:06.120061 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.119665 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78db44bb49-2slcq"] Jul 19 14:59:06.300732 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.299365 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt"] Jul 19 14:59:06.302637 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.302568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.306603 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.306244 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 19 14:59:06.306757 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.306655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-et1t5oa1e4il\"" Jul 19 14:59:06.309134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.307803 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 19 14:59:06.309134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.308146 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 19 14:59:06.309134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.308374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kvxc4\"" Jul 19 14:59:06.309134 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.308575 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 19 14:59:06.310955 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.310631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt"] Jul 19 14:59:06.439434 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-secret-metrics-server-client-certs\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.439434 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7eb8b6b0-eb43-4396-a251-3f7cbe554336-audit-log\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.439652 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-client-ca-bundle\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.439652 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-secret-metrics-server-tls\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.439652 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7eb8b6b0-eb43-4396-a251-3f7cbe554336-metrics-server-audit-profiles\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.439652 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb8b6b0-eb43-4396-a251-3f7cbe554336-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.439860 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.439679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqw2\" (UniqueName: \"kubernetes.io/projected/7eb8b6b0-eb43-4396-a251-3f7cbe554336-kube-api-access-5kqw2\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541147 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7eb8b6b0-eb43-4396-a251-3f7cbe554336-metrics-server-audit-profiles\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb8b6b0-eb43-4396-a251-3f7cbe554336-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqw2\" (UniqueName: \"kubernetes.io/projected/7eb8b6b0-eb43-4396-a251-3f7cbe554336-kube-api-access-5kqw2\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-secret-metrics-server-client-certs\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7eb8b6b0-eb43-4396-a251-3f7cbe554336-audit-log\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541324 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-client-ca-bundle\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.541522 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.541351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-secret-metrics-server-tls\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.542732 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.542098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7eb8b6b0-eb43-4396-a251-3f7cbe554336-metrics-server-audit-profiles\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.542732 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.542429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7eb8b6b0-eb43-4396-a251-3f7cbe554336-audit-log\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.543091 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.543047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7eb8b6b0-eb43-4396-a251-3f7cbe554336-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.544959 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.544912 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-secret-metrics-server-client-certs\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.545234 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.545211 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-secret-metrics-server-tls\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.545730 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.545707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7eb8b6b0-eb43-4396-a251-3f7cbe554336-client-ca-bundle\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.550751 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.550714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqw2\" (UniqueName: \"kubernetes.io/projected/7eb8b6b0-eb43-4396-a251-3f7cbe554336-kube-api-access-5kqw2\") pod \"metrics-server-5cdfb59d7f-kpnwt\" (UID: \"7eb8b6b0-eb43-4396-a251-3f7cbe554336\") " pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.622494 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.622458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:06.770744 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.770634 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl"] Jul 19 14:59:06.773323 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.773297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:06.775905 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.775880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qcpn5\"" Jul 19 14:59:06.776038 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.775880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 19 14:59:06.782521 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.782461 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl"] Jul 19 14:59:06.843980 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.843936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe0a3e92-266d-4de2-86db-4efdf0c82312-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:06.844162 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.844069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8kc\" (UniqueName: \"kubernetes.io/projected/fe0a3e92-266d-4de2-86db-4efdf0c82312-kube-api-access-9m8kc\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:06.945472 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.945439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8kc\" (UniqueName: \"kubernetes.io/projected/fe0a3e92-266d-4de2-86db-4efdf0c82312-kube-api-access-9m8kc\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:06.945633 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.945514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe0a3e92-266d-4de2-86db-4efdf0c82312-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:06.945991 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:06.945701 2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 19 14:59:06.945991 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:06.945776 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe0a3e92-266d-4de2-86db-4efdf0c82312-monitoring-plugin-cert podName:fe0a3e92-266d-4de2-86db-4efdf0c82312 nodeName:}" failed. No retries permitted until 2026-07-19 14:59:07.445754951 +0000 UTC m=+69.306310930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/fe0a3e92-266d-4de2-86db-4efdf0c82312-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-9tppl" (UID: "fe0a3e92-266d-4de2-86db-4efdf0c82312") : secret "monitoring-plugin-cert" not found Jul 19 14:59:06.965457 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:06.955145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8kc\" (UniqueName: \"kubernetes.io/projected/fe0a3e92-266d-4de2-86db-4efdf0c82312-kube-api-access-9m8kc\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:07.056628 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.056543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cd8nd" event={"ID":"b5b7f146-7844-4978-b6ea-e2e61559fe5a","Type":"ContainerStarted","Data":"8d07b6e87a343e869a05431b01ae1d0b21b148c07a4ad85756e5f37e7ec3d00a"} Jul 19 14:59:07.056628 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.056604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cd8nd" event={"ID":"b5b7f146-7844-4978-b6ea-e2e61559fe5a","Type":"ContainerStarted","Data":"c0ba02fcf4e45bc804d9f1e90861eb311fc65eb59fda2fbc3e72a710e928c56a"} Jul 19 14:59:07.058372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.058342 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="107365eda426223c5639c49aeb2ef86ff86df759b2afeb9d27e0b6d2b86fd09c" exitCode=0 Jul 19 14:59:07.058496 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.058431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"107365eda426223c5639c49aeb2ef86ff86df759b2afeb9d27e0b6d2b86fd09c"} Jul 19 14:59:07.061133 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.061020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" event={"ID":"fc6a2f39-a519-474e-889c-28bbfd94bf4e","Type":"ContainerStarted","Data":"77e54eeeebff8d95a7f2b14b148d52bd72176b139eb7b166343e725bdf910e39"} Jul 19 14:59:07.061133 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.061072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" event={"ID":"fc6a2f39-a519-474e-889c-28bbfd94bf4e","Type":"ContainerStarted","Data":"a9e958911cbfe85705798c4f2360fd135273a7b4ebc356f92688da491eaf284c"} Jul 19 14:59:07.061133 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.061100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" event={"ID":"fc6a2f39-a519-474e-889c-28bbfd94bf4e","Type":"ContainerStarted","Data":"1ad38452a6ff12d96f3578c2055fe6d42f1d37fa8cd3692c1f9a7e3f1289e40d"} Jul 19 14:59:07.062611 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.062580 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"75f70d1a788ed8a6c34d6500e05f9ff13928ab0351bdaaa85f620f733a033db1"} Jul 19 14:59:07.064709 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.064631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc8s7" event={"ID":"b9f28b58-145b-4dcb-a3b2-82741e2ad378","Type":"ContainerStarted","Data":"693b35e4bd9922bbad804ba2777694b51f62f2e1e97cbefdc13f5c823e146dd6"} Jul 19 14:59:07.064709 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.064678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc8s7" event={"ID":"b9f28b58-145b-4dcb-a3b2-82741e2ad378","Type":"ContainerStarted","Data":"7e5940786e9c6f2120c7cc4512e905cce990634399d4e2376453abcbb456896c"} Jul 19 14:59:07.067409 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.067379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" event={"ID":"a7367334-c047-4877-a867-b9fd05c218ac","Type":"ContainerStarted","Data":"194f7eeca0e828bda4329b2ad20ac74ab2d54a82c0a491b2b376ca17a7e16fc2"} Jul 19 14:59:07.075338 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.075296 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cd8nd" podStartSLOduration=4.904782289 podStartE2EDuration="6.075282643s" totalImagesPullingTime="1.080414823s" totalInitContainerRuntime="90.085531ms" isStatefulPod=true podCreationTimestamp="2026-07-19 14:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:07.074580064 +0000 UTC m=+68.935136053" watchObservedRunningTime="2026-07-19 14:59:07.075282643 +0000 UTC m=+68.935838622" Jul 19 14:59:07.099695 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.099646 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sl8v4" podStartSLOduration=3.456467572 podStartE2EDuration="6.099627643s" totalImagesPullingTime="2.643160071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:07.098677977 +0000 UTC m=+68.959233968" watchObservedRunningTime="2026-07-19 14:59:07.099627643 +0000 UTC m=+68.960183631" Jul 19 14:59:07.117585 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.117512 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-n9kmq" podStartSLOduration=3.310948991 podStartE2EDuration="6.117492334s" totalImagesPullingTime="2.806543343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:07.116553414 +0000 UTC m=+68.977109402" watchObservedRunningTime="2026-07-19 14:59:07.117492334 +0000 UTC m=+68.978048324" Jul 19 14:59:07.159352 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.159291 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tc8s7" podStartSLOduration=66.864651453 podStartE2EDuration="1m9.159269386s" totalImagesPullingTime="2.294617933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:07.158734997 +0000 UTC m=+69.019290985" watchObservedRunningTime="2026-07-19 14:59:07.159269386 +0000 UTC m=+69.019825369" Jul 19 14:59:07.450626 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.450585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe0a3e92-266d-4de2-86db-4efdf0c82312-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:07.453355 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.453318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe0a3e92-266d-4de2-86db-4efdf0c82312-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9tppl\" (UID: \"fe0a3e92-266d-4de2-86db-4efdf0c82312\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:07.684014 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.683957 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:59:07.684202 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.684047 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:59:07.688349 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.688325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:07.689517 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:07.689493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:59:08.078112 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.078053 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:59:08.130004 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.129949 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff48696c7-9b7m2"] Jul 19 14:59:08.223592 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.223558 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 14:59:08.229319 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.229295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.232285 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232233 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1lr32m5tp3sc3\"" Jul 19 14:59:08.232414 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 19 14:59:08.232595 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232535 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 19 14:59:08.232707 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232236 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 19 14:59:08.232928 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gnrmc\"" Jul 19 14:59:08.232928 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232888 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 19 14:59:08.233072 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.233046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 19 14:59:08.233139 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.233097 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 19 14:59:08.233197 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.232838 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 19 14:59:08.233318 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.233289 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 19 14:59:08.233318 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.233049 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 19 14:59:08.233752 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.233732 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 19 14:59:08.235108 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.235086 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 19 14:59:08.237155 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.237133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 19 14:59:08.237738 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.237718 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 19 14:59:08.243595 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.243574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 14:59:08.245423 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.245401 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 19 14:59:08.364057 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.363975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364057 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364057 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364369 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364369 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364369 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-config\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364369 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364369 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t962v\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-kube-api-access-t962v\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364369 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.364742 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.364731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466004 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.465960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466188 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-config\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t962v\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-kube-api-access-t962v\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466531 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466997 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466997 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466997 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466997 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.466997 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.466872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.467521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.468309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.468978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.470344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.470853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.471293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.473321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.473702 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.473659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.475864 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.475799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.476227 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.476158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.477965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.476930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.477965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.477042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.477965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.477111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.477965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.477336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.477965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.477474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-config\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.477965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.477928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.478668 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.478622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t962v\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-kube-api-access-t962v\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.479225 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.479152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:08.547965 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:08.547697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:09.175304 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.174199 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75bd49568-gg5gs"] Jul 19 14:59:09.182507 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.182414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.190197 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.190115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bd49568-gg5gs"] Jul 19 14:59:09.276947 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.276915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k554\" (UniqueName: \"kubernetes.io/projected/75f32d4d-5a86-4935-9fe9-778b5231cbbd-kube-api-access-6k554\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.276947 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.276958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-service-ca\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.277380 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.276978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-trusted-ca-bundle\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.277380 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.277061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-serving-cert\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.277380 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.277114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-config\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.277380 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.277149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-oauth-serving-cert\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.277380 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.277168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-oauth-config\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378238 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k554\" (UniqueName: \"kubernetes.io/projected/75f32d4d-5a86-4935-9fe9-778b5231cbbd-kube-api-access-6k554\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378457 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-service-ca\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378457 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-trusted-ca-bundle\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378457 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-serving-cert\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378457 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-config\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378457 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-oauth-serving-cert\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.378733 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.378517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-oauth-config\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.379308 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.379076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-service-ca\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.379409 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.379356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-config\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.379468 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.379416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-oauth-serving-cert\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.379595 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.379559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-trusted-ca-bundle\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.381495 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.381472 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-oauth-config\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.381578 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.381560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-serving-cert\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.390769 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.390750 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k554\" (UniqueName: \"kubernetes.io/projected/75f32d4d-5a86-4935-9fe9-778b5231cbbd-kube-api-access-6k554\") pod \"console-75bd49568-gg5gs\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:09.498299 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:09.498183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:10.112817 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:10.112748 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bd49568-gg5gs"] Jul 19 14:59:10.147147 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:10.147115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl"] Jul 19 14:59:10.187684 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:10.187653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 14:59:10.310784 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:10.310754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt"] Jul 19 14:59:10.451465 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:10.451241 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0a3e92_266d_4de2_86db_4efdf0c82312.slice/crio-28e0211924f3ecce42100ea61b1aed98f00b31d9730cdda20daef6ea85864565 WatchSource:0}: Error finding container 28e0211924f3ecce42100ea61b1aed98f00b31d9730cdda20daef6ea85864565: Status 404 returned error can't find the container with id 28e0211924f3ecce42100ea61b1aed98f00b31d9730cdda20daef6ea85864565 Jul 19 14:59:10.452057 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:10.452034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb8b6b0_eb43_4396_a251_3f7cbe554336.slice/crio-430169e8f35dea1a6095dd9df2dd86a579afb50b82e8b75389a0aea0e6f0e6a5 WatchSource:0}: Error finding container 430169e8f35dea1a6095dd9df2dd86a579afb50b82e8b75389a0aea0e6f0e6a5: Status 404 returned error can't find the container with id 430169e8f35dea1a6095dd9df2dd86a579afb50b82e8b75389a0aea0e6f0e6a5 Jul 19 14:59:10.453110 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:10.453089 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f32d4d_5a86_4935_9fe9_778b5231cbbd.slice/crio-1fbb0275f57c6deb009b806726edd11b37ed8cddea2ede86ce4d623aacbfa5d3 WatchSource:0}: Error finding container 1fbb0275f57c6deb009b806726edd11b37ed8cddea2ede86ce4d623aacbfa5d3: Status 404 returned error can't find the container with id 1fbb0275f57c6deb009b806726edd11b37ed8cddea2ede86ce4d623aacbfa5d3 Jul 19 14:59:10.454004 ip-10-0-142-31 kubenswrapper[2579]: W0719 14:59:10.453877 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b094c8b_9977_4e39_b1f4_882e86262f85.slice/crio-c05a0134e81c723d01c1f9ddb4d56042a4bedd61075b8192a266a1d29fe7a880 WatchSource:0}: Error finding container c05a0134e81c723d01c1f9ddb4d56042a4bedd61075b8192a266a1d29fe7a880: Status 404 returned error can't find the container with id c05a0134e81c723d01c1f9ddb4d56042a4bedd61075b8192a266a1d29fe7a880 Jul 19 14:59:11.091325 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.091215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mg8z6" event={"ID":"9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21","Type":"ContainerStarted","Data":"89aa55e373d7bf3c97ee311fbce83b4a7a73ff901e002c028b37d8f5562aedf4"} Jul 19 14:59:11.095901 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.095867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"b9e40eb713d514ab688f3320fe9ac7ae99ed841c9d95ba4fe3a701c7616f0195"} Jul 19 14:59:11.096062 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.095950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"a4edd3388cd186337133700e615a3262a7b1d61fceacd4c05d823d684e38cc0f"} Jul 19 14:59:11.096062 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.095993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"e894f922ee347c126fef2f123217d9909f78d31c59b6103314f00d07329dd44a"} Jul 19 14:59:11.096062 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.096034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"64dc407a0b10abca72fc78199b144ca085eae25e8786625ac0f272c8cf92b071"} Jul 19 14:59:11.096237 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.096074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"3e1c0d2158a6c7b2ae53150f82eacbf9b5fe8d2c60938287e8bb4c8f893f4e2f"} Jul 19 14:59:11.098500 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.098469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"e6203b41b20315ca300386eaf16bb1b5217791e8b61cb88dafa8ff6ce6f1ee18"} Jul 19 14:59:11.098609 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.098541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"b094677f518be89aa8dd5995e9a9862722b68745a7872c066a77bc573039864a"} Jul 19 14:59:11.098609 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.098585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"2274446e8ab8b389e0c27a9371ed344e0b75a97b63fc7bc866e7cab76aaabf5a"} Jul 19 14:59:11.100833 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.100784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd49568-gg5gs" event={"ID":"75f32d4d-5a86-4935-9fe9-778b5231cbbd","Type":"ContainerStarted","Data":"8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8"} Jul 19 14:59:11.100833 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.100831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd49568-gg5gs" event={"ID":"75f32d4d-5a86-4935-9fe9-778b5231cbbd","Type":"ContainerStarted","Data":"1fbb0275f57c6deb009b806726edd11b37ed8cddea2ede86ce4d623aacbfa5d3"} Jul 19 14:59:11.102322 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.102294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" event={"ID":"fe0a3e92-266d-4de2-86db-4efdf0c82312","Type":"ContainerStarted","Data":"28e0211924f3ecce42100ea61b1aed98f00b31d9730cdda20daef6ea85864565"} Jul 19 14:59:11.103686 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.103650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" event={"ID":"7eb8b6b0-eb43-4396-a251-3f7cbe554336","Type":"ContainerStarted","Data":"430169e8f35dea1a6095dd9df2dd86a579afb50b82e8b75389a0aea0e6f0e6a5"} Jul 19 14:59:11.106526 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.105591 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" exitCode=0 Jul 19 14:59:11.106526 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.105692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} Jul 19 14:59:11.106526 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.105752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"c05a0134e81c723d01c1f9ddb4d56042a4bedd61075b8192a266a1d29fe7a880"} Jul 19 14:59:11.107868 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.107827 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mg8z6" podStartSLOduration=66.803121683 podStartE2EDuration="1m13.107815444s" totalImagesPullingTime="6.304693761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:11.106036489 +0000 UTC m=+72.966592469" watchObservedRunningTime="2026-07-19 14:59:11.107815444 +0000 UTC m=+72.968371431" Jul 19 14:59:11.108560 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.108509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5jldc" event={"ID":"c8c066bb-94e3-4586-8d1f-cb9821fcce88","Type":"ContainerStarted","Data":"634aa14727051f9763bdb6086ca1452891f06f0b84629034699616bb2ebe713a"} Jul 19 14:59:11.108727 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.108689 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:59:11.123456 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.123391 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75bd49568-gg5gs" podStartSLOduration=2.1233669 podStartE2EDuration="2.1233669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:11.123276244 +0000 UTC m=+72.983832234" watchObservedRunningTime="2026-07-19 14:59:11.1233669 +0000 UTC m=+72.983922888" Jul 19 14:59:11.165712 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:11.165655 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5jldc" podStartSLOduration=67.265789782 podStartE2EDuration="1m13.165632963s" totalImagesPullingTime="5.899843181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:11.163912448 +0000 UTC m=+73.024468435" watchObservedRunningTime="2026-07-19 14:59:11.165632963 +0000 UTC m=+73.026188952" Jul 19 14:59:13.124350 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.123672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerStarted","Data":"f4d6528d32a24e52f75a2440b9f69718d01127ab7e74bfc297193455fa4ea030"} Jul 19 14:59:13.131381 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.131348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"91125223c23042a4a32300847a54e661dd9ef312bf63b30db30425ee81db4de8"} Jul 19 14:59:13.131537 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.131427 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"975cf72ab75b7bdd9374a51a78668d0f838678f0323593ebe46515b39e835501"} Jul 19 14:59:13.131537 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.131472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" event={"ID":"0c17fba3-b20b-4128-b203-d96fa6cb5aea","Type":"ContainerStarted","Data":"ae1bdfa5d73b5870913fe87e14f4b8f3fc982fc357b47c93fb9df539a15fa7bf"} Jul 19 14:59:13.131537 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.131516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:13.133061 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.133033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" event={"ID":"fe0a3e92-266d-4de2-86db-4efdf0c82312","Type":"ContainerStarted","Data":"737792dafc82fa8db084caf7f10944560cea8d1530f3b3ab561952c8cef63a32"} Jul 19 14:59:13.133333 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.133289 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:13.135203 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.135162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" event={"ID":"7eb8b6b0-eb43-4396-a251-3f7cbe554336","Type":"ContainerStarted","Data":"8b09a4310663157d28c961633adc13117ba0506c7c1febdb8f0efa4a28ee9e4f"} Jul 19 14:59:13.139575 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.139555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" Jul 19 14:59:13.155415 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.155365 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.721583486 podStartE2EDuration="10.155349722s" totalImagesPullingTime="7.387793553s" totalInitContainerRuntime="45.972683ms" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:13.154495521 +0000 UTC m=+75.015051544" watchObservedRunningTime="2026-07-19 14:59:13.155349722 +0000 UTC m=+75.015905709" Jul 19 14:59:13.176132 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.176072 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" podStartSLOduration=5.137253329 podStartE2EDuration="7.176052425s" totalImagesPullingTime="2.038799096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:13.175459737 +0000 UTC m=+75.036015727" watchObservedRunningTime="2026-07-19 14:59:13.176052425 +0000 UTC m=+75.036608414" Jul 19 14:59:13.194568 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.194512 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9tppl" podStartSLOduration=5.151467452 podStartE2EDuration="7.194494252s" totalImagesPullingTime="2.0430268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:13.194104889 +0000 UTC m=+75.054660880" watchObservedRunningTime="2026-07-19 14:59:13.194494252 +0000 UTC m=+75.055050239" Jul 19 14:59:13.216051 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:13.215995 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" podStartSLOduration=3.507073639 podStartE2EDuration="9.215972347s" totalImagesPullingTime="5.708898708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:13.215703261 +0000 UTC m=+75.076259250" watchObservedRunningTime="2026-07-19 14:59:13.215972347 +0000 UTC m=+75.076528336" Jul 19 14:59:15.149533 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.149448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} Jul 19 14:59:15.149892 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.149567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} Jul 19 14:59:15.149892 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.149609 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} Jul 19 14:59:15.149892 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.149652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} Jul 19 14:59:15.149892 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.149696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} Jul 19 14:59:15.149892 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.149734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerStarted","Data":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} Jul 19 14:59:15.180023 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:15.179973 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.63191726 podStartE2EDuration="7.179955884s" totalImagesPullingTime="3.512822835s" totalInitContainerRuntime="35.215789ms" isStatefulPod=false podCreationTimestamp="2026-07-19 14:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:15.178071839 +0000 UTC m=+77.038627837" watchObservedRunningTime="2026-07-19 14:59:15.179955884 +0000 UTC m=+77.040511913" Jul 19 14:59:18.548136 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:18.548101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 14:59:19.147695 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:19.147665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78db44bb49-2slcq" Jul 19 14:59:19.498798 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:19.498717 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:19.498798 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:19.498771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:19.503436 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:19.503411 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:20.176406 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:20.176282 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 14:59:20.222979 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:20.222947 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc5dd7fc6-ksgfb"] Jul 19 14:59:22.639386 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:22.639354 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gpj89_f449b873-72ee-4366-97ab-65026aba1100/serve-healthcheck-canary/0.log" Jul 19 14:59:26.623047 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:26.623016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:26.623516 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:26.623068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:33.157317 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:33.157250 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff48696c7-9b7m2" podUID="2e41acde-c4b9-43da-874f-3a46bf6d1b43" containerName="console" containerID="cri-o://20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b" gracePeriod=15 Jul 19 14:59:33.944885 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:33.944854 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:59:34.017838 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.017804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-oauth-config\") pod \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " Jul 19 14:59:34.018091 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018075 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-oauth-serving-cert\") pod \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " Jul 19 14:59:34.018202 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018187 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-config\") pod \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " Jul 19 14:59:34.018486 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018451 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9k49\" (UniqueName: \"kubernetes.io/projected/2e41acde-c4b9-43da-874f-3a46bf6d1b43-kube-api-access-s9k49\") pod \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " Jul 19 14:59:34.018602 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018497 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-service-ca\") pod \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " Jul 19 14:59:34.018602 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018549 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-serving-cert\") pod \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\" (UID: \"2e41acde-c4b9-43da-874f-3a46bf6d1b43\") " Jul 19 14:59:34.018862 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018799 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e41acde-c4b9-43da-874f-3a46bf6d1b43" (UID: "2e41acde-c4b9-43da-874f-3a46bf6d1b43"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:34.018862 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.018822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-config" (OuterVolumeSpecName: "console-config") pod "2e41acde-c4b9-43da-874f-3a46bf6d1b43" (UID: "2e41acde-c4b9-43da-874f-3a46bf6d1b43"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:34.019128 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.019052 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e41acde-c4b9-43da-874f-3a46bf6d1b43" (UID: "2e41acde-c4b9-43da-874f-3a46bf6d1b43"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:34.021023 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.020993 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e41acde-c4b9-43da-874f-3a46bf6d1b43" (UID: "2e41acde-c4b9-43da-874f-3a46bf6d1b43"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:34.021138 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.021112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e41acde-c4b9-43da-874f-3a46bf6d1b43" (UID: "2e41acde-c4b9-43da-874f-3a46bf6d1b43"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:34.021564 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.021538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e41acde-c4b9-43da-874f-3a46bf6d1b43-kube-api-access-s9k49" (OuterVolumeSpecName: "kube-api-access-s9k49") pod "2e41acde-c4b9-43da-874f-3a46bf6d1b43" (UID: "2e41acde-c4b9-43da-874f-3a46bf6d1b43"). InnerVolumeSpecName "kube-api-access-s9k49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:34.119723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.119633 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9k49\" (UniqueName: \"kubernetes.io/projected/2e41acde-c4b9-43da-874f-3a46bf6d1b43-kube-api-access-s9k49\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:34.119723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.119661 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-service-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:34.119723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.119670 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:34.119723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.119679 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-oauth-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:34.119723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.119688 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-oauth-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:34.119723 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.119696 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e41acde-c4b9-43da-874f-3a46bf6d1b43-console-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:34.248031 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.247989 2579 generic.go:356] "Generic (PLEG): container finished" podID="2e41acde-c4b9-43da-874f-3a46bf6d1b43" containerID="20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b" exitCode=0 Jul 19 14:59:34.248499 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.248148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff48696c7-9b7m2" event={"ID":"2e41acde-c4b9-43da-874f-3a46bf6d1b43","Type":"ContainerDied","Data":"20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b"} Jul 19 14:59:34.248499 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.248237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff48696c7-9b7m2" event={"ID":"2e41acde-c4b9-43da-874f-3a46bf6d1b43","Type":"ContainerDied","Data":"c11de5843f62d7272dd4de3e25d0794de9a9a28db87187fc1047469de686ade7"} Jul 19 14:59:34.248499 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.248317 2579 scope.go:117] "RemoveContainer" containerID="20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b" Jul 19 14:59:34.248642 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.248514 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff48696c7-9b7m2" Jul 19 14:59:34.260629 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.260598 2579 scope.go:117] "RemoveContainer" containerID="20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b" Jul 19 14:59:34.260980 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:34.260952 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b\": container with ID starting with 20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b not found: ID does not exist" containerID="20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b" Jul 19 14:59:34.261100 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.260987 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b"} err="failed to get container status \"20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b\": rpc error: code = NotFound desc = could not find container \"20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b\": container with ID starting with 20029236b33aedb252ef72f6e8158c0b110d94663a36ffc2543bc2a387c6b34b not found: ID does not exist" Jul 19 14:59:34.275372 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.275296 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff48696c7-9b7m2"] Jul 19 14:59:34.280494 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.280458 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff48696c7-9b7m2"] Jul 19 14:59:34.619106 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:34.619071 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e41acde-c4b9-43da-874f-3a46bf6d1b43" path="/var/lib/kubelet/pods/2e41acde-c4b9-43da-874f-3a46bf6d1b43/volumes" Jul 19 14:59:42.116642 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:42.116613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5jldc" Jul 19 14:59:45.256694 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.256632 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cc5dd7fc6-ksgfb" podUID="41aa65a5-39ba-489c-9108-bfd928bf5f17" containerName="console" containerID="cri-o://ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0" gracePeriod=15 Jul 19 14:59:45.494412 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.494389 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:59:45.628787 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-config\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.628969 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-oauth-serving-cert\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.628969 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-serving-cert\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.628969 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-oauth-config\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.628969 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-service-ca\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.628969 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms6h8\" (UniqueName: \"kubernetes.io/projected/41aa65a5-39ba-489c-9108-bfd928bf5f17-kube-api-access-ms6h8\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.629218 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.628975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-trusted-ca-bundle\") pod \"41aa65a5-39ba-489c-9108-bfd928bf5f17\" (UID: \"41aa65a5-39ba-489c-9108-bfd928bf5f17\") " Jul 19 14:59:45.629298 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.629206 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-config" (OuterVolumeSpecName: "console-config") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:45.629417 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.629384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:45.629577 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.629528 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:45.629577 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.629429 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-service-ca" (OuterVolumeSpecName: "service-ca") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:45.629744 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.629537 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:45.631131 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.631106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41aa65a5-39ba-489c-9108-bfd928bf5f17-kube-api-access-ms6h8" (OuterVolumeSpecName: "kube-api-access-ms6h8") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "kube-api-access-ms6h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:45.631207 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.631123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:45.631207 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.631166 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41aa65a5-39ba-489c-9108-bfd928bf5f17" (UID: "41aa65a5-39ba-489c-9108-bfd928bf5f17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:45.730949 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.730912 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:45.730949 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.730943 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41aa65a5-39ba-489c-9108-bfd928bf5f17-console-oauth-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:45.730949 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.730957 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-service-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:45.731172 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.730970 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6h8\" (UniqueName: \"kubernetes.io/projected/41aa65a5-39ba-489c-9108-bfd928bf5f17-kube-api-access-ms6h8\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:45.731172 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.730982 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:45.731172 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:45.730993 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41aa65a5-39ba-489c-9108-bfd928bf5f17-oauth-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 14:59:46.302782 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.302739 2579 generic.go:356] "Generic (PLEG): container finished" podID="41aa65a5-39ba-489c-9108-bfd928bf5f17" containerID="ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0" exitCode=0 Jul 19 14:59:46.303220 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.302801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc5dd7fc6-ksgfb" Jul 19 14:59:46.303220 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.302803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5dd7fc6-ksgfb" event={"ID":"41aa65a5-39ba-489c-9108-bfd928bf5f17","Type":"ContainerDied","Data":"ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0"} Jul 19 14:59:46.303220 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.302913 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc5dd7fc6-ksgfb" event={"ID":"41aa65a5-39ba-489c-9108-bfd928bf5f17","Type":"ContainerDied","Data":"ea0162bbfdcd7512d79319f1cb97f9751fb3a904e7e823127cc2dd292070a497"} Jul 19 14:59:46.303220 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.302933 2579 scope.go:117] "RemoveContainer" containerID="ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0" Jul 19 14:59:46.311666 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.311650 2579 scope.go:117] "RemoveContainer" containerID="ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0" Jul 19 14:59:46.311949 ip-10-0-142-31 kubenswrapper[2579]: E0719 14:59:46.311924 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0\": container with ID starting with ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0 not found: ID does not exist" containerID="ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0" Jul 19 14:59:46.311994 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.311959 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0"} err="failed to get container status \"ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0\": rpc error: code = NotFound desc = could not find container \"ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0\": container with ID starting with ac80f3ca611ac5de1db5d95f5c38d23e8b9df724eeb65d553c004148936106c0 not found: ID does not exist" Jul 19 14:59:46.326230 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.326207 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc5dd7fc6-ksgfb"] Jul 19 14:59:46.329512 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.329480 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cc5dd7fc6-ksgfb"] Jul 19 14:59:46.612001 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.611926 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41aa65a5-39ba-489c-9108-bfd928bf5f17" path="/var/lib/kubelet/pods/41aa65a5-39ba-489c-9108-bfd928bf5f17/volumes" Jul 19 14:59:46.628105 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.628082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 14:59:46.631860 ip-10-0-142-31 kubenswrapper[2579]: I0719 14:59:46.631838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5cdfb59d7f-kpnwt" Jul 19 15:00:08.548923 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:08.548879 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:08.568926 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:08.568901 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:09.415794 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:09.415764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:22.243875 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.243838 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 15:00:22.244582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.244121 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="alertmanager" containerID="cri-o://3e1c0d2158a6c7b2ae53150f82eacbf9b5fe8d2c60938287e8bb4c8f893f4e2f" gracePeriod=120 Jul 19 15:00:22.244582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.244187 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-metric" containerID="cri-o://b9e40eb713d514ab688f3320fe9ac7ae99ed841c9d95ba4fe3a701c7616f0195" gracePeriod=120 Jul 19 15:00:22.244582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.244217 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="prom-label-proxy" containerID="cri-o://f4d6528d32a24e52f75a2440b9f69718d01127ab7e74bfc297193455fa4ea030" gracePeriod=120 Jul 19 15:00:22.244582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.244279 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-web" containerID="cri-o://e894f922ee347c126fef2f123217d9909f78d31c59b6103314f00d07329dd44a" gracePeriod=120 Jul 19 15:00:22.244582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.244305 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="config-reloader" containerID="cri-o://64dc407a0b10abca72fc78199b144ca085eae25e8786625ac0f272c8cf92b071" gracePeriod=120 Jul 19 15:00:22.244582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.244358 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy" containerID="cri-o://a4edd3388cd186337133700e615a3262a7b1d61fceacd4c05d823d684e38cc0f" gracePeriod=120 Jul 19 15:00:22.454629 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454593 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="f4d6528d32a24e52f75a2440b9f69718d01127ab7e74bfc297193455fa4ea030" exitCode=0 Jul 19 15:00:22.454629 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454620 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="b9e40eb713d514ab688f3320fe9ac7ae99ed841c9d95ba4fe3a701c7616f0195" exitCode=0 Jul 19 15:00:22.454629 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454630 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="a4edd3388cd186337133700e615a3262a7b1d61fceacd4c05d823d684e38cc0f" exitCode=0 Jul 19 15:00:22.454629 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454638 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="64dc407a0b10abca72fc78199b144ca085eae25e8786625ac0f272c8cf92b071" exitCode=0 Jul 19 15:00:22.454889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454646 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="3e1c0d2158a6c7b2ae53150f82eacbf9b5fe8d2c60938287e8bb4c8f893f4e2f" exitCode=0 Jul 19 15:00:22.454889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"f4d6528d32a24e52f75a2440b9f69718d01127ab7e74bfc297193455fa4ea030"} Jul 19 15:00:22.454889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"b9e40eb713d514ab688f3320fe9ac7ae99ed841c9d95ba4fe3a701c7616f0195"} Jul 19 15:00:22.454889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"a4edd3388cd186337133700e615a3262a7b1d61fceacd4c05d823d684e38cc0f"} Jul 19 15:00:22.454889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"64dc407a0b10abca72fc78199b144ca085eae25e8786625ac0f272c8cf92b071"} Jul 19 15:00:22.454889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:22.454805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"3e1c0d2158a6c7b2ae53150f82eacbf9b5fe8d2c60938287e8bb4c8f893f4e2f"} Jul 19 15:00:23.462520 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.462486 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerID="e894f922ee347c126fef2f123217d9909f78d31c59b6103314f00d07329dd44a" exitCode=0 Jul 19 15:00:23.462956 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.462554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"e894f922ee347c126fef2f123217d9909f78d31c59b6103314f00d07329dd44a"} Jul 19 15:00:23.495242 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.495219 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:23.548724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548683 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-main-db\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.548899 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psj6l\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-kube-api-access-psj6l\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.548899 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548819 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-volume\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.548899 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548870 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-metrics-client-ca\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.548899 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548898 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-out\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-main-tls\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548966 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.549022 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-web\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.549058 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-web-config\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.549091 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-trusted-ca-bundle\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549394 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.549120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549394 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.549149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-tls-assets\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.549394 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.548960 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:00:23.550493 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.549938 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-cluster-tls-config\") pod \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\" (UID: \"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f\") " Jul 19 15:00:23.550493 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.550308 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-main-db\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.550677 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.550645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:23.550835 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.550805 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:23.552273 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.552234 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:00:23.553296 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.553237 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-kube-api-access-psj6l" (OuterVolumeSpecName: "kube-api-access-psj6l") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "kube-api-access-psj6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:00:23.553803 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.553725 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-out" (OuterVolumeSpecName: "config-out") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:00:23.554525 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.554471 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.554660 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.554585 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.554844 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.554682 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.554844 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.554822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-volume" (OuterVolumeSpecName: "config-volume") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.555112 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.555078 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.559992 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.559969 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.568313 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.568280 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-web-config" (OuterVolumeSpecName: "web-config") pod "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" (UID: "ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:23.651421 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651377 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psj6l\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-kube-api-access-psj6l\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651421 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651415 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-volume\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651421 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651429 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-metrics-client-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651441 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-config-out\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651455 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-main-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651471 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651486 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651500 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-web-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651513 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651528 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651541 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-tls-assets\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:23.651650 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:23.651555 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f-cluster-tls-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:24.469548 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.469509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f","Type":"ContainerDied","Data":"8e0eb799b07701eac4829f2984e44d1c191a02f40e3c68136b83345bbee668ab"} Jul 19 15:00:24.469950 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.469588 2579 scope.go:117] "RemoveContainer" containerID="f4d6528d32a24e52f75a2440b9f69718d01127ab7e74bfc297193455fa4ea030" Jul 19 15:00:24.469950 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.469521 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.478009 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.477985 2579 scope.go:117] "RemoveContainer" containerID="b9e40eb713d514ab688f3320fe9ac7ae99ed841c9d95ba4fe3a701c7616f0195" Jul 19 15:00:24.485593 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.485575 2579 scope.go:117] "RemoveContainer" containerID="a4edd3388cd186337133700e615a3262a7b1d61fceacd4c05d823d684e38cc0f" Jul 19 15:00:24.493587 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.493561 2579 scope.go:117] "RemoveContainer" containerID="e894f922ee347c126fef2f123217d9909f78d31c59b6103314f00d07329dd44a" Jul 19 15:00:24.494505 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.494481 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 15:00:24.502811 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.498300 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 15:00:24.502972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.502908 2579 scope.go:117] "RemoveContainer" containerID="64dc407a0b10abca72fc78199b144ca085eae25e8786625ac0f272c8cf92b071" Jul 19 15:00:24.510316 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.510298 2579 scope.go:117] "RemoveContainer" containerID="3e1c0d2158a6c7b2ae53150f82eacbf9b5fe8d2c60938287e8bb4c8f893f4e2f" Jul 19 15:00:24.517563 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.517548 2579 scope.go:117] "RemoveContainer" containerID="107365eda426223c5639c49aeb2ef86ff86df759b2afeb9d27e0b6d2b86fd09c" Jul 19 15:00:24.525479 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525459 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 15:00:24.525773 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525762 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="alertmanager" Jul 19 15:00:24.525813 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525775 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="alertmanager" Jul 19 15:00:24.525813 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525788 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="init-config-reloader" Jul 19 15:00:24.525813 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525794 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="init-config-reloader" Jul 19 15:00:24.525813 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525801 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41aa65a5-39ba-489c-9108-bfd928bf5f17" containerName="console" Jul 19 15:00:24.525813 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525807 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41aa65a5-39ba-489c-9108-bfd928bf5f17" containerName="console" Jul 19 15:00:24.525813 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525813 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525826 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="config-reloader" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525831 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="config-reloader" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525838 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-metric" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525843 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-metric" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525852 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="prom-label-proxy" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525857 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="prom-label-proxy" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525867 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-web" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525872 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-web" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525881 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e41acde-c4b9-43da-874f-3a46bf6d1b43" containerName="console" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525886 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e41acde-c4b9-43da-874f-3a46bf6d1b43" containerName="console" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525926 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="alertmanager" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525934 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="config-reloader" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525940 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-metric" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525946 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41aa65a5-39ba-489c-9108-bfd928bf5f17" containerName="console" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525953 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e41acde-c4b9-43da-874f-3a46bf6d1b43" containerName="console" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525959 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525964 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="prom-label-proxy" Jul 19 15:00:24.525989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.525971 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" containerName="kube-rbac-proxy-web" Jul 19 15:00:24.529396 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.529380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.532347 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 19 15:00:24.532443 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532322 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 19 15:00:24.532608 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532586 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 19 15:00:24.532665 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532587 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2gnrw\"" Jul 19 15:00:24.532665 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 19 15:00:24.532665 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 19 15:00:24.532807 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 19 15:00:24.532807 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532587 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 19 15:00:24.532807 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.532637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 19 15:00:24.537168 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.537147 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 19 15:00:24.542252 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.542232 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 15:00:24.560050 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560158 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560158 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560158 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-config-out\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560484 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-web-config\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560484 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560484 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560484 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcp9q\" (UniqueName: \"kubernetes.io/projected/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-kube-api-access-bcp9q\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.560484 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.560434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.612235 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.612157 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f" path="/var/lib/kubelet/pods/ea2ef8aa-4065-4d79-8c6e-27f2c3e2d25f/volumes" Jul 19 15:00:24.661204 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661517 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661517 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661517 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-config-out\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661517 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661517 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-web-config\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661749 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661749 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661749 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcp9q\" (UniqueName: \"kubernetes.io/projected/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-kube-api-access-bcp9q\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.661749 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.661610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.663962 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.662481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.663962 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.663088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.663962 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.663403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664317 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664404 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664787 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664787 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-web-config\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664787 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664771 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664787 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664782 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.664950 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.664846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.665705 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.665687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-config-out\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.666415 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.666394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.671925 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.671902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcp9q\" (UniqueName: \"kubernetes.io/projected/6e487a7c-f0e2-4661-bff6-c350fdac8bb7-kube-api-access-bcp9q\") pod \"alertmanager-main-0\" (UID: \"6e487a7c-f0e2-4661-bff6-c350fdac8bb7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.839279 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.839225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 19 15:00:24.978662 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:24.978635 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 19 15:00:24.981104 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:00:24.981078 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e487a7c_f0e2_4661_bff6_c350fdac8bb7.slice/crio-dd3ca30efab6c248e522cb025a00bab12db0952823e8cb07fd1835579592ec31 WatchSource:0}: Error finding container dd3ca30efab6c248e522cb025a00bab12db0952823e8cb07fd1835579592ec31: Status 404 returned error can't find the container with id dd3ca30efab6c248e522cb025a00bab12db0952823e8cb07fd1835579592ec31 Jul 19 15:00:25.475974 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:25.475938 2579 generic.go:356] "Generic (PLEG): container finished" podID="6e487a7c-f0e2-4661-bff6-c350fdac8bb7" containerID="90bd230baf73f8636b9bc45deaf221077b71f35a6f953eb370105e87e361e76b" exitCode=0 Jul 19 15:00:25.476357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:25.476007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerDied","Data":"90bd230baf73f8636b9bc45deaf221077b71f35a6f953eb370105e87e361e76b"} Jul 19 15:00:25.476357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:25.476071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"dd3ca30efab6c248e522cb025a00bab12db0952823e8cb07fd1835579592ec31"} Jul 19 15:00:26.484300 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.484240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"ec96c792eb411793a23b6e1acb9b23cef984cfab7d8fb5ef29811e5cc7ef51f6"} Jul 19 15:00:26.484782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.484336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"cc2596282d683e1a10b5712e75347812b893b3bbe2accc24d8af74e6460eac80"} Jul 19 15:00:26.484782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.484378 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"9af333ccff3f0495216c180b384366dd930581278fd7c8995e6a2b210c26aed9"} Jul 19 15:00:26.484782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.484418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"98333a70641919082b5c7785ca13917debf699aa301c83a756e860c9fa58ab81"} Jul 19 15:00:26.484782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.484461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"1c65b626f9d6fd08967320670eeb89f01b881e7ad7cd42b72c22341de3dd1554"} Jul 19 15:00:26.484782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.484495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e487a7c-f0e2-4661-bff6-c350fdac8bb7","Type":"ContainerStarted","Data":"7ccbdb19925ef23d42efae4aa390b920605fe4912f10b69e97a205f6721ddeea"} Jul 19 15:00:26.512058 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.511998 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.495936153 podStartE2EDuration="2.511981379s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.045226ms" isStatefulPod=false podCreationTimestamp="2026-07-19 15:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:00:26.51163812 +0000 UTC m=+148.372194109" watchObservedRunningTime="2026-07-19 15:00:26.511981379 +0000 UTC m=+148.372537367" Jul 19 15:00:26.598160 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598120 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 15:00:26.598526 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598488 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="prometheus" containerID="cri-o://bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" gracePeriod=600 Jul 19 15:00:26.598683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598496 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-web" containerID="cri-o://3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" gracePeriod=600 Jul 19 15:00:26.598683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598512 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy" containerID="cri-o://3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" gracePeriod=600 Jul 19 15:00:26.598683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598512 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="config-reloader" containerID="cri-o://afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" gracePeriod=600 Jul 19 15:00:26.598683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598594 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" gracePeriod=600 Jul 19 15:00:26.598683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.598496 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="thanos-sidecar" containerID="cri-o://eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" gracePeriod=600 Jul 19 15:00:26.856558 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.856534 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:26.986042 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986008 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-db\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986042 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-2\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986073 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-config-out\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-kubelet-serving-ca-bundle\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986113 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-web-config\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986131 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-tls-assets\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986146 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986165 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-0\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-serving-certs-ca-bundle\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986215 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t962v\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-kube-api-access-t962v\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986244 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-1\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-metrics-client-certs\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986362 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-trusted-ca-bundle\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986407 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-grpc-tls\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986462 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986497 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-tls\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986519 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-kube-rbac-proxy\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-metrics-client-ca\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986593 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-config\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.986731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986617 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-thanos-prometheus-http-client-file\") pod \"5b094c8b-9977-4e39-b1f4-882e86262f85\" (UID: \"5b094c8b-9977-4e39-b1f4-882e86262f85\") " Jul 19 15:00:26.987403 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.987502 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.986543 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.987502 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.987095 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:00:26.987502 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.987370 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.987502 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.987475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.987758 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.987617 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.988811 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.988443 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.988811 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.988472 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:00:26.989325 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.989163 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-kube-api-access-t962v" (OuterVolumeSpecName: "kube-api-access-t962v") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "kube-api-access-t962v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:00:26.989428 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.989394 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:00:26.989566 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.989538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.989931 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.989844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-config-out" (OuterVolumeSpecName: "config-out") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:00:26.990472 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.990447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.990986 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.990961 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.991303 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.991277 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.991576 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.991556 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-config" (OuterVolumeSpecName: "config") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.991662 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.991634 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.991725 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.991691 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.991790 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.991766 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:26.999519 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:26.999499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-web-config" (OuterVolumeSpecName: "web-config") pod "5b094c8b-9977-4e39-b1f4-882e86262f85" (UID: "5b094c8b-9977-4e39-b1f4-882e86262f85"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:00:27.088104 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088009 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088104 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088045 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088104 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088060 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-kube-rbac-proxy\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088104 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088073 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-metrics-client-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088104 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088088 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088104 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088101 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088115 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-db\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088128 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088139 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b094c8b-9977-4e39-b1f4-882e86262f85-config-out\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088151 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088162 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-web-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088177 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-tls-assets\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088188 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088200 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088212 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088224 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t962v\" (UniqueName: \"kubernetes.io/projected/5b094c8b-9977-4e39-b1f4-882e86262f85-kube-api-access-t962v\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088236 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088250 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-metrics-client-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088290 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b094c8b-9977-4e39-b1f4-882e86262f85-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.088469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.088302 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b094c8b-9977-4e39-b1f4-882e86262f85-secret-grpc-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:00:27.491478 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491445 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" exitCode=0 Jul 19 15:00:27.491478 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491470 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" exitCode=0 Jul 19 15:00:27.491478 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491476 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" exitCode=0 Jul 19 15:00:27.491478 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491482 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" exitCode=0 Jul 19 15:00:27.491478 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491487 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" exitCode=0 Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491492 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" exitCode=0 Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491521 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491661 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} Jul 19 15:00:27.492107 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.491723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b094c8b-9977-4e39-b1f4-882e86262f85","Type":"ContainerDied","Data":"c05a0134e81c723d01c1f9ddb4d56042a4bedd61075b8192a266a1d29fe7a880"} Jul 19 15:00:27.499494 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.499474 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.506411 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.506394 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.513461 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.513443 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.516859 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.516835 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 15:00:27.521452 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.521435 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.521999 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.521975 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 15:00:27.528382 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.528359 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.535551 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.535531 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.542364 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.542339 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.542619 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.542600 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.542672 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.542631 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} err="failed to get container status \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" Jul 19 15:00:27.542672 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.542653 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.542884 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.542864 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.542947 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.542893 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} err="failed to get container status \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" Jul 19 15:00:27.542947 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.542915 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.543160 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.543144 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.543194 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543163 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} err="failed to get container status \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" Jul 19 15:00:27.543194 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543178 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.543406 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.543389 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.543453 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543410 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} err="failed to get container status \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" Jul 19 15:00:27.543453 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543423 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.543613 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.543598 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.543672 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543619 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} err="failed to get container status \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" Jul 19 15:00:27.543672 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543638 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.543840 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.543826 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.543874 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543842 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} err="failed to get container status \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" Jul 19 15:00:27.543874 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.543854 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.544069 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:00:27.544052 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.544142 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.544075 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} err="failed to get container status \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" Jul 19 15:00:27.544142 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.544096 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.544720 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.544661 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} err="failed to get container status \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" Jul 19 15:00:27.544824 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.544722 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.545074 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545055 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} err="failed to get container status \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" Jul 19 15:00:27.545160 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545077 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.545495 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545469 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} err="failed to get container status \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" Jul 19 15:00:27.545580 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545495 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.545751 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545730 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} err="failed to get container status \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" Jul 19 15:00:27.545825 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545753 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.546014 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.545993 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} err="failed to get container status \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" Jul 19 15:00:27.546014 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.546012 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.546335 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.546314 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} err="failed to get container status \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" Jul 19 15:00:27.546335 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.546335 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.546719 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547147 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="thanos-sidecar" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547162 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="thanos-sidecar" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547174 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547183 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547198 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-web" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547206 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-web" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547223 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="init-config-reloader" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547231 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="init-config-reloader" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547245 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="prometheus" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547272 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="prometheus" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547289 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="config-reloader" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547297 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="config-reloader" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547306 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-thanos" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547314 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-thanos" Jul 19 15:00:27.547474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547469 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-thanos" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547491 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="prometheus" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547502 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy-web" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547553 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="config-reloader" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547576 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="thanos-sidecar" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.547585 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" containerName="kube-rbac-proxy" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.549171 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} err="failed to get container status \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" Jul 19 15:00:27.549760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.549195 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.550046 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.549950 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} err="failed to get container status \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" Jul 19 15:00:27.550046 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.549974 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.550393 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.550366 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} err="failed to get container status \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" Jul 19 15:00:27.550393 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.550392 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.550635 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.550610 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} err="failed to get container status \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" Jul 19 15:00:27.550709 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.550648 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.550883 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.550865 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} err="failed to get container status \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" Jul 19 15:00:27.550947 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.550883 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.551084 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551066 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} err="failed to get container status \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" Jul 19 15:00:27.551137 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551084 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.551280 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551249 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} err="failed to get container status \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" Jul 19 15:00:27.551375 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551281 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.551454 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551440 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} err="failed to get container status \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" Jul 19 15:00:27.551500 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551454 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.551649 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551627 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} err="failed to get container status \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" Jul 19 15:00:27.551688 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551652 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.551825 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.551949 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551911 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} err="failed to get container status \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" Jul 19 15:00:27.551949 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.551927 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.552232 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552174 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} err="failed to get container status \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" Jul 19 15:00:27.552232 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552204 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.552437 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552419 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} err="failed to get container status \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" Jul 19 15:00:27.552497 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552439 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.552681 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552661 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} err="failed to get container status \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" Jul 19 15:00:27.552731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552684 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.552894 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552875 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} err="failed to get container status \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" Jul 19 15:00:27.552944 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.552894 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.553120 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553097 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} err="failed to get container status \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" Jul 19 15:00:27.553165 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553123 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.553416 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553394 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} err="failed to get container status \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" Jul 19 15:00:27.553494 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553418 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.553668 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553649 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} err="failed to get container status \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" Jul 19 15:00:27.553734 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553669 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.553938 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553915 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} err="failed to get container status \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" Jul 19 15:00:27.553993 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.553938 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.554178 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554155 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} err="failed to get container status \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" Jul 19 15:00:27.554224 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554180 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.554496 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554474 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} err="failed to get container status \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" Jul 19 15:00:27.554496 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554494 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.554677 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554528 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 19 15:00:27.554733 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554712 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} err="failed to get container status \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" Jul 19 15:00:27.554733 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554728 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.554981 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} err="failed to get container status \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555007 2579 scope.go:117] "RemoveContainer" containerID="9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555075 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555076 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555112 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555127 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 19 15:00:27.555237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555329 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1lr32m5tp3sc3\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555333 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555420 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d"} err="failed to get container status \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": rpc error: code = NotFound desc = could not find container \"9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d\": container with ID starting with 9610e000358de238f8f6112870f37b36dc80b64b47296f3b8acd67cb2ee59a8d not found: ID does not exist" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555451 2579 scope.go:117] "RemoveContainer" containerID="3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555557 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555567 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555574 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 19 15:00:27.555669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555605 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gnrmc\"" Jul 19 15:00:27.556059 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555803 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36"} err="failed to get container status \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": rpc error: code = NotFound desc = could not find container \"3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36\": container with ID starting with 3bcb6d9bcc92f93cfea3993e7542953c3dffa7ef9d411ceb2fb45023d96ebf36 not found: ID does not exist" Jul 19 15:00:27.556059 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.555827 2579 scope.go:117] "RemoveContainer" containerID="3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6" Jul 19 15:00:27.556152 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.556128 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6"} err="failed to get container status \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": rpc error: code = NotFound desc = could not find container \"3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6\": container with ID starting with 3501833b135aa2a6485191bc6cc4799470c0f4855d1a5a905627402306e6d1e6 not found: ID does not exist" Jul 19 15:00:27.556205 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.556155 2579 scope.go:117] "RemoveContainer" containerID="eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a" Jul 19 15:00:27.556523 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.556500 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a"} err="failed to get container status \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": rpc error: code = NotFound desc = could not find container \"eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a\": container with ID starting with eb34e25722da6beef08c64f4f75da97af0d561dbee9f18b10e94d7620a10277a not found: ID does not exist" Jul 19 15:00:27.556615 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.556525 2579 scope.go:117] "RemoveContainer" containerID="afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee" Jul 19 15:00:27.556827 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.556806 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee"} err="failed to get container status \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": rpc error: code = NotFound desc = could not find container \"afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee\": container with ID starting with afc794058b28f3df8257092465e29346e285af53b14fd2474bfdbb615ef992ee not found: ID does not exist" Jul 19 15:00:27.556901 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.556828 2579 scope.go:117] "RemoveContainer" containerID="bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc" Jul 19 15:00:27.557079 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.557056 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc"} err="failed to get container status \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": rpc error: code = NotFound desc = could not find container \"bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc\": container with ID starting with bd9a4f256fdd828143cd20f6efc17fbe03d2b74fd728c0850237cf3aeda75fcc not found: ID does not exist" Jul 19 15:00:27.557155 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.557082 2579 scope.go:117] "RemoveContainer" containerID="49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb" Jul 19 15:00:27.557367 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.557343 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb"} err="failed to get container status \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": rpc error: code = NotFound desc = could not find container \"49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb\": container with ID starting with 49f79f4b9b1f9ca35a490179b861ac2624dda367a3cc1519cbc38cbb20c04bbb not found: ID does not exist" Jul 19 15:00:27.558117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.558099 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 19 15:00:27.559879 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.559834 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 19 15:00:27.564365 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.564343 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 15:00:27.592306 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592306 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592512 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592512 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592623 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7de23c68-0731-4681-8251-88ad538ab4d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592623 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592623 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldcs\" (UniqueName: \"kubernetes.io/projected/7de23c68-0731-4681-8251-88ad538ab4d2-kube-api-access-bldcs\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592864 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592864 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.592864 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7de23c68-0731-4681-8251-88ad538ab4d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593005 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593005 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.592948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-config\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593108 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.593034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593108 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.593098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593188 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.593127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593188 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.593144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.593287 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.593192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.693990 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.693951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.693990 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.693993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694193 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694242 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7de23c68-0731-4681-8251-88ad538ab4d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694482 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldcs\" (UniqueName: \"kubernetes.io/projected/7de23c68-0731-4681-8251-88ad538ab4d2-kube-api-access-bldcs\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.694482 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695220 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.694817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695220 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695220 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7de23c68-0731-4681-8251-88ad538ab4d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-config\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695818 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695818 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695818 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.695959 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.695887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.696732 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.696707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.697157 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.697126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.697240 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.697175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.697334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.697297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.697334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.697296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.698051 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.697904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.698396 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.698241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.698396 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.698358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.698629 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.698581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7de23c68-0731-4681-8251-88ad538ab4d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.698629 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.698582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7de23c68-0731-4681-8251-88ad538ab4d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.699610 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.699585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-config\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.699686 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.699669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.699811 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.699787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.699866 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.699849 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.700441 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.700423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7de23c68-0731-4681-8251-88ad538ab4d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.700773 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.700755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7de23c68-0731-4681-8251-88ad538ab4d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.703191 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.703175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldcs\" (UniqueName: \"kubernetes.io/projected/7de23c68-0731-4681-8251-88ad538ab4d2-kube-api-access-bldcs\") pod \"prometheus-k8s-0\" (UID: \"7de23c68-0731-4681-8251-88ad538ab4d2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.863081 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.862987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:27.999367 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:27.999334 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 19 15:00:28.002208 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:00:28.002182 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de23c68_0731_4681_8251_88ad538ab4d2.slice/crio-c3196854d2d9236eca0a93048a59a0e6e2069dd2e1163ae0377c7cc15be24117 WatchSource:0}: Error finding container c3196854d2d9236eca0a93048a59a0e6e2069dd2e1163ae0377c7cc15be24117: Status 404 returned error can't find the container with id c3196854d2d9236eca0a93048a59a0e6e2069dd2e1163ae0377c7cc15be24117 Jul 19 15:00:28.496605 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:28.496569 2579 generic.go:356] "Generic (PLEG): container finished" podID="7de23c68-0731-4681-8251-88ad538ab4d2" containerID="d7fd40cbacef86bcb23403d99493a5ab15056581b692e8219bbf989f1b1bca9a" exitCode=0 Jul 19 15:00:28.497087 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:28.496707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerDied","Data":"d7fd40cbacef86bcb23403d99493a5ab15056581b692e8219bbf989f1b1bca9a"} Jul 19 15:00:28.497087 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:28.496793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"c3196854d2d9236eca0a93048a59a0e6e2069dd2e1163ae0377c7cc15be24117"} Jul 19 15:00:28.615506 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:28.615185 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b094c8b-9977-4e39-b1f4-882e86262f85" path="/var/lib/kubelet/pods/5b094c8b-9977-4e39-b1f4-882e86262f85/volumes" Jul 19 15:00:29.506733 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.506697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"4d8ded675793b6f277a8ae77088a499da9ff3420b64b2156fed3b3849226c70d"} Jul 19 15:00:29.507153 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.506787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"7fec6b4717d1c00316e28232641f93764b96d814a205eac64f2b2419648d1e25"} Jul 19 15:00:29.507153 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.506814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"fb47745e1dc1db5da82fed2eb597a4976370e4c54a4189d4b18e538a4ecf586e"} Jul 19 15:00:29.507153 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.506839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"f8fec54757a0e614c9e830eca86fd4203c0f1d4317071c31bd5aee8f1ec6f48b"} Jul 19 15:00:29.507153 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.506863 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"80bddfbc9665119d1a1b0df7ad7c0a76833a6448ad3e5d3815b2fab36a980c1c"} Jul 19 15:00:29.507153 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.506888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7de23c68-0731-4681-8251-88ad538ab4d2","Type":"ContainerStarted","Data":"875e59b61997b7cab1e617aedabf3dfd9e2bc070bb003cdf62ad660d15f70680"} Jul 19 15:00:29.535111 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:29.535049 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.51667393 podStartE2EDuration="2.535030889s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.356959ms" isStatefulPod=false podCreationTimestamp="2026-07-19 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:00:29.533192999 +0000 UTC m=+151.393748986" watchObservedRunningTime="2026-07-19 15:00:29.535030889 +0000 UTC m=+151.395586877" Jul 19 15:00:31.410253 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.410216 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c6cb7747f-w4f52"] Jul 19 15:00:31.413578 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.413556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.424341 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.424311 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6cb7747f-w4f52"] Jul 19 15:00:31.529550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-serving-cert\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.529550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-service-ca\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.529756 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65cz\" (UniqueName: \"kubernetes.io/projected/08889ccb-7663-4443-b16f-aa03552d7294-kube-api-access-z65cz\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.529756 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-oauth-config\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.529829 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-oauth-serving-cert\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.529829 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-trusted-ca-bundle\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.529889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.529824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-console-config\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631338 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-oauth-serving-cert\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631514 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-trusted-ca-bundle\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631514 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-console-config\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631514 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-serving-cert\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631514 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-service-ca\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631514 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65cz\" (UniqueName: \"kubernetes.io/projected/08889ccb-7663-4443-b16f-aa03552d7294-kube-api-access-z65cz\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.631514 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.631482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-oauth-config\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.632222 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.632192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-service-ca\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.632377 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.632231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-oauth-serving-cert\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.632377 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.632245 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-console-config\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.632459 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.632393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-trusted-ca-bundle\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.633942 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.633917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-serving-cert\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.634601 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.634585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-oauth-config\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.639356 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.639330 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65cz\" (UniqueName: \"kubernetes.io/projected/08889ccb-7663-4443-b16f-aa03552d7294-kube-api-access-z65cz\") pod \"console-c6cb7747f-w4f52\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.724990 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.724890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:31.865877 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:31.865847 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6cb7747f-w4f52"] Jul 19 15:00:31.868861 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:00:31.868831 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08889ccb_7663_4443_b16f_aa03552d7294.slice/crio-b6c53cf73c6ac18dce1343490a8b458feab736c3453353a132b4644d05168575 WatchSource:0}: Error finding container b6c53cf73c6ac18dce1343490a8b458feab736c3453353a132b4644d05168575: Status 404 returned error can't find the container with id b6c53cf73c6ac18dce1343490a8b458feab736c3453353a132b4644d05168575 Jul 19 15:00:32.522832 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:32.522792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6cb7747f-w4f52" event={"ID":"08889ccb-7663-4443-b16f-aa03552d7294","Type":"ContainerStarted","Data":"9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813"} Jul 19 15:00:32.523228 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:32.522856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6cb7747f-w4f52" event={"ID":"08889ccb-7663-4443-b16f-aa03552d7294","Type":"ContainerStarted","Data":"b6c53cf73c6ac18dce1343490a8b458feab736c3453353a132b4644d05168575"} Jul 19 15:00:32.540291 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:32.540232 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c6cb7747f-w4f52" podStartSLOduration=1.540215903 podStartE2EDuration="1.540215903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:00:32.539010231 +0000 UTC m=+154.399566216" watchObservedRunningTime="2026-07-19 15:00:32.540215903 +0000 UTC m=+154.400771890" Jul 19 15:00:32.863855 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:32.863776 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:00:41.725902 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:41.725860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:41.726373 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:41.725921 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:41.730722 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:41.730705 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:42.569473 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:42.569448 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:00:42.615859 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:00:42.615828 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75bd49568-gg5gs"] Jul 19 15:01:07.641679 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.641626 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75bd49568-gg5gs" podUID="75f32d4d-5a86-4935-9fe9-778b5231cbbd" containerName="console" containerID="cri-o://8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8" gracePeriod=15 Jul 19 15:01:07.880858 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.880835 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 15:01:07.955714 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955626 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-config\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.955714 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955664 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-oauth-config\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.955714 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955691 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-serving-cert\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.955996 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955738 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-trusted-ca-bundle\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.955996 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k554\" (UniqueName: \"kubernetes.io/projected/75f32d4d-5a86-4935-9fe9-778b5231cbbd-kube-api-access-6k554\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.955996 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955782 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-service-ca\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.955996 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.955821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-oauth-serving-cert\") pod \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\" (UID: \"75f32d4d-5a86-4935-9fe9-778b5231cbbd\") " Jul 19 15:01:07.956202 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.956122 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-config" (OuterVolumeSpecName: "console-config") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:01:07.956331 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.956306 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:01:07.956395 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.956332 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-service-ca" (OuterVolumeSpecName: "service-ca") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:01:07.956395 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.956388 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:01:07.958003 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.957979 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f32d4d-5a86-4935-9fe9-778b5231cbbd-kube-api-access-6k554" (OuterVolumeSpecName: "kube-api-access-6k554") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "kube-api-access-6k554". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:01:07.958421 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.958392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:01:07.958506 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:07.958471 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "75f32d4d-5a86-4935-9fe9-778b5231cbbd" (UID: "75f32d4d-5a86-4935-9fe9-778b5231cbbd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:01:08.056531 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056492 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-oauth-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.056531 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056525 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.056531 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056534 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-oauth-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.056762 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056544 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f32d4d-5a86-4935-9fe9-778b5231cbbd-console-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.056762 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056553 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.056762 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056561 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k554\" (UniqueName: \"kubernetes.io/projected/75f32d4d-5a86-4935-9fe9-778b5231cbbd-kube-api-access-6k554\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.056762 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.056570 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75f32d4d-5a86-4935-9fe9-778b5231cbbd-service-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:01:08.671135 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.671101 2579 generic.go:356] "Generic (PLEG): container finished" podID="75f32d4d-5a86-4935-9fe9-778b5231cbbd" containerID="8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8" exitCode=0 Jul 19 15:01:08.671551 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.671156 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd49568-gg5gs" Jul 19 15:01:08.671551 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.671168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd49568-gg5gs" event={"ID":"75f32d4d-5a86-4935-9fe9-778b5231cbbd","Type":"ContainerDied","Data":"8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8"} Jul 19 15:01:08.671551 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.671227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd49568-gg5gs" event={"ID":"75f32d4d-5a86-4935-9fe9-778b5231cbbd","Type":"ContainerDied","Data":"1fbb0275f57c6deb009b806726edd11b37ed8cddea2ede86ce4d623aacbfa5d3"} Jul 19 15:01:08.671551 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.671245 2579 scope.go:117] "RemoveContainer" containerID="8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8" Jul 19 15:01:08.679581 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.679566 2579 scope.go:117] "RemoveContainer" containerID="8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8" Jul 19 15:01:08.679839 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:01:08.679816 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8\": container with ID starting with 8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8 not found: ID does not exist" containerID="8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8" Jul 19 15:01:08.679934 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.679845 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8"} err="failed to get container status \"8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8\": rpc error: code = NotFound desc = could not find container \"8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8\": container with ID starting with 8e7163f0e5518f88621e1914e324facc2612bcc888b53d7dbd7f75b808dd4ee8 not found: ID does not exist" Jul 19 15:01:08.688495 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.688474 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75bd49568-gg5gs"] Jul 19 15:01:08.692190 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:08.692168 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75bd49568-gg5gs"] Jul 19 15:01:10.612314 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:10.612283 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f32d4d-5a86-4935-9fe9-778b5231cbbd" path="/var/lib/kubelet/pods/75f32d4d-5a86-4935-9fe9-778b5231cbbd/volumes" Jul 19 15:01:27.863957 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:27.863917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:01:27.879420 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:27.879388 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:01:28.764587 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:28.764560 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 19 15:01:39.670625 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.670591 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765f765db9-4p95c"] Jul 19 15:01:39.671030 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.670909 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f32d4d-5a86-4935-9fe9-778b5231cbbd" containerName="console" Jul 19 15:01:39.671030 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.670920 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f32d4d-5a86-4935-9fe9-778b5231cbbd" containerName="console" Jul 19 15:01:39.671030 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.670978 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f32d4d-5a86-4935-9fe9-778b5231cbbd" containerName="console" Jul 19 15:01:39.675025 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.675004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.683908 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.683877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765f765db9-4p95c"] Jul 19 15:01:39.722800 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.722764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-oauth-config\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.722800 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.722800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vzq\" (UniqueName: \"kubernetes.io/projected/05644a33-7bec-4bd2-9a59-044bf33a510f-kube-api-access-96vzq\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.722972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.722822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-serving-cert\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.722972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.722950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-service-ca\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.723041 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.722996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-console-config\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.723041 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.723025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-trusted-ca-bundle\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.723131 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.723044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-oauth-serving-cert\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823531 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-service-ca\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-console-config\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-trusted-ca-bundle\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-oauth-serving-cert\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-oauth-config\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vzq\" (UniqueName: \"kubernetes.io/projected/05644a33-7bec-4bd2-9a59-044bf33a510f-kube-api-access-96vzq\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.823711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.823649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-serving-cert\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.824401 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.824376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-oauth-serving-cert\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.824530 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.824400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-service-ca\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.824530 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.824440 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-console-config\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.824826 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.824806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-trusted-ca-bundle\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.826034 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.826011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-serving-cert\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.826137 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.826121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-oauth-config\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.831333 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.831314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vzq\" (UniqueName: \"kubernetes.io/projected/05644a33-7bec-4bd2-9a59-044bf33a510f-kube-api-access-96vzq\") pod \"console-765f765db9-4p95c\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:39.984699 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:39.984607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:40.114703 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:40.114594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765f765db9-4p95c"] Jul 19 15:01:40.117020 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:01:40.116994 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05644a33_7bec_4bd2_9a59_044bf33a510f.slice/crio-f23a57dfb81e273daf5480d8583d64a847316914ae1f186e96afb48557ab5f8b WatchSource:0}: Error finding container f23a57dfb81e273daf5480d8583d64a847316914ae1f186e96afb48557ab5f8b: Status 404 returned error can't find the container with id f23a57dfb81e273daf5480d8583d64a847316914ae1f186e96afb48557ab5f8b Jul 19 15:01:40.798376 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:40.798340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f765db9-4p95c" event={"ID":"05644a33-7bec-4bd2-9a59-044bf33a510f","Type":"ContainerStarted","Data":"ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942"} Jul 19 15:01:40.798780 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:40.798395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f765db9-4p95c" event={"ID":"05644a33-7bec-4bd2-9a59-044bf33a510f","Type":"ContainerStarted","Data":"f23a57dfb81e273daf5480d8583d64a847316914ae1f186e96afb48557ab5f8b"} Jul 19 15:01:40.815112 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:40.815061 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765f765db9-4p95c" podStartSLOduration=1.815040694 podStartE2EDuration="1.815040694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:01:40.814618308 +0000 UTC m=+222.675174298" watchObservedRunningTime="2026-07-19 15:01:40.815040694 +0000 UTC m=+222.675596681" Jul 19 15:01:49.985312 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:49.985239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:49.985723 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:49.985343 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:49.990000 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:49.989974 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:50.846400 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:50.846370 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:01:50.889120 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:01:50.889088 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6cb7747f-w4f52"] Jul 19 15:02:15.917749 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:15.917689 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c6cb7747f-w4f52" podUID="08889ccb-7663-4443-b16f-aa03552d7294" containerName="console" containerID="cri-o://9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813" gracePeriod=15 Jul 19 15:02:16.161295 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.161271 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:02:16.351594 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351515 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-service-ca\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.351594 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z65cz\" (UniqueName: \"kubernetes.io/projected/08889ccb-7663-4443-b16f-aa03552d7294-kube-api-access-z65cz\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.351594 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-oauth-config\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.351849 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351609 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-serving-cert\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.351849 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-oauth-serving-cert\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.351849 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-trusted-ca-bundle\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.351849 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.351699 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-console-config\") pod \"08889ccb-7663-4443-b16f-aa03552d7294\" (UID: \"08889ccb-7663-4443-b16f-aa03552d7294\") " Jul 19 15:02:16.352293 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.352064 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-service-ca" (OuterVolumeSpecName: "service-ca") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:02:16.352293 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.352195 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:02:16.352293 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.352205 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-console-config" (OuterVolumeSpecName: "console-config") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:02:16.352293 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.352238 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:02:16.353777 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.353749 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:02:16.353905 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.353806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:02:16.353905 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.353835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08889ccb-7663-4443-b16f-aa03552d7294-kube-api-access-z65cz" (OuterVolumeSpecName: "kube-api-access-z65cz") pod "08889ccb-7663-4443-b16f-aa03552d7294" (UID: "08889ccb-7663-4443-b16f-aa03552d7294"). InnerVolumeSpecName "kube-api-access-z65cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:02:16.458940 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458902 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-service-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.458940 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458933 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65cz\" (UniqueName: \"kubernetes.io/projected/08889ccb-7663-4443-b16f-aa03552d7294-kube-api-access-z65cz\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.458940 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458944 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-oauth-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.459216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458955 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08889ccb-7663-4443-b16f-aa03552d7294-console-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.459216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458963 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-oauth-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.459216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458971 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.459216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.458979 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08889ccb-7663-4443-b16f-aa03552d7294-console-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:02:16.947783 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.947751 2579 generic.go:356] "Generic (PLEG): container finished" podID="08889ccb-7663-4443-b16f-aa03552d7294" containerID="9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813" exitCode=0 Jul 19 15:02:16.948177 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.947810 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6cb7747f-w4f52" Jul 19 15:02:16.948177 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.947837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6cb7747f-w4f52" event={"ID":"08889ccb-7663-4443-b16f-aa03552d7294","Type":"ContainerDied","Data":"9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813"} Jul 19 15:02:16.948177 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.947889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6cb7747f-w4f52" event={"ID":"08889ccb-7663-4443-b16f-aa03552d7294","Type":"ContainerDied","Data":"b6c53cf73c6ac18dce1343490a8b458feab736c3453353a132b4644d05168575"} Jul 19 15:02:16.948177 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.947910 2579 scope.go:117] "RemoveContainer" containerID="9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813" Jul 19 15:02:16.956207 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.956191 2579 scope.go:117] "RemoveContainer" containerID="9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813" Jul 19 15:02:16.956500 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:02:16.956482 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813\": container with ID starting with 9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813 not found: ID does not exist" containerID="9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813" Jul 19 15:02:16.956566 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.956508 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813"} err="failed to get container status \"9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813\": rpc error: code = NotFound desc = could not find container \"9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813\": container with ID starting with 9ffbbbf0244dc0541ae90771a9b263f2d184b026229011692190c3418a4c7813 not found: ID does not exist" Jul 19 15:02:16.964886 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.964855 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6cb7747f-w4f52"] Jul 19 15:02:16.968652 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:16.968629 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c6cb7747f-w4f52"] Jul 19 15:02:18.612108 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:18.612080 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08889ccb-7663-4443-b16f-aa03552d7294" path="/var/lib/kubelet/pods/08889ccb-7663-4443-b16f-aa03552d7294/volumes" Jul 19 15:02:58.495680 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:58.495649 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:02:58.498827 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:58.498802 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:02:58.502136 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:02:58.502112 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 19 15:03:09.334575 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.334539 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6"] Jul 19 15:03:09.336867 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.335093 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08889ccb-7663-4443-b16f-aa03552d7294" containerName="console" Jul 19 15:03:09.336867 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.335110 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08889ccb-7663-4443-b16f-aa03552d7294" containerName="console" Jul 19 15:03:09.336867 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.335195 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08889ccb-7663-4443-b16f-aa03552d7294" containerName="console" Jul 19 15:03:09.338626 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.338607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.341229 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.341206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7c9td\"" Jul 19 15:03:09.341376 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.341244 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 19 15:03:09.342381 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.342344 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 19 15:03:09.344010 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.343988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6"] Jul 19 15:03:09.411775 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.411741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e21956-6278-43c6-b40c-383892d486e1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lf8w6\" (UID: \"33e21956-6278-43c6-b40c-383892d486e1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.411966 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.411870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snj2l\" (UniqueName: \"kubernetes.io/projected/33e21956-6278-43c6-b40c-383892d486e1-kube-api-access-snj2l\") pod \"cert-manager-cainjector-559cdd8c87-lf8w6\" (UID: \"33e21956-6278-43c6-b40c-383892d486e1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.438449 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.438417 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qwgpr"] Jul 19 15:03:09.441597 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.441581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.444274 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.444245 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-kc8rn\"" Jul 19 15:03:09.451315 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.451295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qwgpr"] Jul 19 15:03:09.512972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.512935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhkq\" (UniqueName: \"kubernetes.io/projected/1751502a-a81e-49b1-bbb6-48c5668efff8-kube-api-access-wmhkq\") pod \"cert-manager-webhook-6486ffd997-qwgpr\" (UID: \"1751502a-a81e-49b1-bbb6-48c5668efff8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.513162 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.512984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snj2l\" (UniqueName: \"kubernetes.io/projected/33e21956-6278-43c6-b40c-383892d486e1-kube-api-access-snj2l\") pod \"cert-manager-cainjector-559cdd8c87-lf8w6\" (UID: \"33e21956-6278-43c6-b40c-383892d486e1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.513162 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.513004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1751502a-a81e-49b1-bbb6-48c5668efff8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qwgpr\" (UID: \"1751502a-a81e-49b1-bbb6-48c5668efff8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.513162 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.513059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e21956-6278-43c6-b40c-383892d486e1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lf8w6\" (UID: \"33e21956-6278-43c6-b40c-383892d486e1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.520954 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.520929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33e21956-6278-43c6-b40c-383892d486e1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lf8w6\" (UID: \"33e21956-6278-43c6-b40c-383892d486e1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.521091 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.521055 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snj2l\" (UniqueName: \"kubernetes.io/projected/33e21956-6278-43c6-b40c-383892d486e1-kube-api-access-snj2l\") pod \"cert-manager-cainjector-559cdd8c87-lf8w6\" (UID: \"33e21956-6278-43c6-b40c-383892d486e1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.614441 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.614363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhkq\" (UniqueName: \"kubernetes.io/projected/1751502a-a81e-49b1-bbb6-48c5668efff8-kube-api-access-wmhkq\") pod \"cert-manager-webhook-6486ffd997-qwgpr\" (UID: \"1751502a-a81e-49b1-bbb6-48c5668efff8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.614441 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.614408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1751502a-a81e-49b1-bbb6-48c5668efff8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qwgpr\" (UID: \"1751502a-a81e-49b1-bbb6-48c5668efff8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.622160 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.622117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1751502a-a81e-49b1-bbb6-48c5668efff8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qwgpr\" (UID: \"1751502a-a81e-49b1-bbb6-48c5668efff8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.622296 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.622273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhkq\" (UniqueName: \"kubernetes.io/projected/1751502a-a81e-49b1-bbb6-48c5668efff8-kube-api-access-wmhkq\") pod \"cert-manager-webhook-6486ffd997-qwgpr\" (UID: \"1751502a-a81e-49b1-bbb6-48c5668efff8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.665889 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.665856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" Jul 19 15:03:09.750612 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.750579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:09.793561 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.793354 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6"] Jul 19 15:03:09.796361 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:03:09.796337 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e21956_6278_43c6_b40c_383892d486e1.slice/crio-3a953a128003cca2f9150184343b76bd5841ea38c30af2e7f19c5e0bd38f663d WatchSource:0}: Error finding container 3a953a128003cca2f9150184343b76bd5841ea38c30af2e7f19c5e0bd38f663d: Status 404 returned error can't find the container with id 3a953a128003cca2f9150184343b76bd5841ea38c30af2e7f19c5e0bd38f663d Jul 19 15:03:09.798357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.798328 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:03:09.880109 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:09.880086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qwgpr"] Jul 19 15:03:09.882039 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:03:09.882007 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1751502a_a81e_49b1_bbb6_48c5668efff8.slice/crio-99b071f60f87415f9a7b53c64e421aae405b3ecb4500386f031b1abecc6e3789 WatchSource:0}: Error finding container 99b071f60f87415f9a7b53c64e421aae405b3ecb4500386f031b1abecc6e3789: Status 404 returned error can't find the container with id 99b071f60f87415f9a7b53c64e421aae405b3ecb4500386f031b1abecc6e3789 Jul 19 15:03:10.156544 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:10.156434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" event={"ID":"1751502a-a81e-49b1-bbb6-48c5668efff8","Type":"ContainerStarted","Data":"99b071f60f87415f9a7b53c64e421aae405b3ecb4500386f031b1abecc6e3789"} Jul 19 15:03:10.157751 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:10.157728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" event={"ID":"33e21956-6278-43c6-b40c-383892d486e1","Type":"ContainerStarted","Data":"3a953a128003cca2f9150184343b76bd5841ea38c30af2e7f19c5e0bd38f663d"} Jul 19 15:03:13.176169 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:13.176131 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" event={"ID":"33e21956-6278-43c6-b40c-383892d486e1","Type":"ContainerStarted","Data":"8c07dbcc8ba2bd908aa418fee3dcd7e0fa57b34ebee13be351926adec7eabd5a"} Jul 19 15:03:13.177857 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:13.177833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" event={"ID":"1751502a-a81e-49b1-bbb6-48c5668efff8","Type":"ContainerStarted","Data":"f21e25fae36196940f6846b625dada825e2e03c86823e4c21b24dfd863be4ce5"} Jul 19 15:03:13.177982 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:13.177948 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:13.192841 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:13.192793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lf8w6" podStartSLOduration=1.028815711 podStartE2EDuration="4.192776504s" totalImagesPullingTime="3.163960793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:13.190973007 +0000 UTC m=+315.051528996" watchObservedRunningTime="2026-07-19 15:03:13.192776504 +0000 UTC m=+315.053332497" Jul 19 15:03:13.208742 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:13.208639 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" podStartSLOduration=1.1191602330000001 podStartE2EDuration="4.20861826s" totalImagesPullingTime="3.089458027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:13.20714867 +0000 UTC m=+315.067704659" watchObservedRunningTime="2026-07-19 15:03:13.20861826 +0000 UTC m=+315.069174250" Jul 19 15:03:19.184196 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:19.184167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qwgpr" Jul 19 15:03:46.244814 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.244780 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5"] Jul 19 15:03:46.293357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.293326 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5"] Jul 19 15:03:46.293519 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.293424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.296541 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.296518 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 19 15:03:46.297795 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.297743 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 19 15:03:46.297795 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.297789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 19 15:03:46.297985 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.297802 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 19 15:03:46.297985 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.297902 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 19 15:03:46.297985 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.297977 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wtfmc\"" Jul 19 15:03:46.323011 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.322984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-cert\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.323131 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.323016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-metrics-cert\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.323131 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.323081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snjw\" (UniqueName: \"kubernetes.io/projected/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-kube-api-access-5snjw\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.323206 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.323144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-manager-config\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.424527 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.424491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-cert\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.424527 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.424528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-metrics-cert\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.424734 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.424576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snjw\" (UniqueName: \"kubernetes.io/projected/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-kube-api-access-5snjw\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.424734 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.424618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-manager-config\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.425291 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.425250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-manager-config\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.427027 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.427003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-cert\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.427121 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.427047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-metrics-cert\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.432851 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.432830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snjw\" (UniqueName: \"kubernetes.io/projected/fbb79277-c8e8-4dbb-8cd4-0574150d6cc5-kube-api-access-5snjw\") pod \"lws-controller-manager-57c99ccc67-9lxl5\" (UID: \"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.603218 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.603134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:46.742807 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:46.742784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5"] Jul 19 15:03:46.745062 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:03:46.745034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb79277_c8e8_4dbb_8cd4_0574150d6cc5.slice/crio-d89866131a467854542e79e89ecaac1ecebf1fd2221f0f5d05a27f51c57f3e63 WatchSource:0}: Error finding container d89866131a467854542e79e89ecaac1ecebf1fd2221f0f5d05a27f51c57f3e63: Status 404 returned error can't find the container with id d89866131a467854542e79e89ecaac1ecebf1fd2221f0f5d05a27f51c57f3e63 Jul 19 15:03:47.321843 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:47.321809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" event={"ID":"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5","Type":"ContainerStarted","Data":"d89866131a467854542e79e89ecaac1ecebf1fd2221f0f5d05a27f51c57f3e63"} Jul 19 15:03:49.333071 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:49.333034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" event={"ID":"fbb79277-c8e8-4dbb-8cd4-0574150d6cc5","Type":"ContainerStarted","Data":"20d765a182576c77e5208437a98ce87eb83dbefb579aa57334bd04ec94bec806"} Jul 19 15:03:49.333496 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:49.333173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:03:49.353324 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:49.353277 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" podStartSLOduration=0.874322478 podStartE2EDuration="3.353247969s" totalImagesPullingTime="2.478925491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:49.350864609 +0000 UTC m=+351.211420598" watchObservedRunningTime="2026-07-19 15:03:49.353247969 +0000 UTC m=+351.213803957" Jul 19 15:03:58.776918 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.776883 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v"] Jul 19 15:03:58.782248 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.782225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.785334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.785305 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 19 15:03:58.785724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.785348 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vwrkx\"" Jul 19 15:03:58.785724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.785586 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 19 15:03:58.792619 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.787610 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 19 15:03:58.792619 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.787803 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 19 15:03:58.802913 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.802889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v"] Jul 19 15:03:58.834904 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.834869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e511cb-2cec-4ee1-9965-89b08ca34aaf-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.835311 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.835187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02e511cb-2cec-4ee1-9965-89b08ca34aaf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.838071 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.838051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02e511cb-2cec-4ee1-9965-89b08ca34aaf-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.838243 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.838224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czm6q\" (UniqueName: \"kubernetes.io/projected/02e511cb-2cec-4ee1-9965-89b08ca34aaf-kube-api-access-czm6q\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.838445 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.838428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02e511cb-2cec-4ee1-9965-89b08ca34aaf-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.939430 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.939391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02e511cb-2cec-4ee1-9965-89b08ca34aaf-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.939621 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.939455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e511cb-2cec-4ee1-9965-89b08ca34aaf-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.939621 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.939475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02e511cb-2cec-4ee1-9965-89b08ca34aaf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.939621 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.939493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02e511cb-2cec-4ee1-9965-89b08ca34aaf-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.939621 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.939515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czm6q\" (UniqueName: \"kubernetes.io/projected/02e511cb-2cec-4ee1-9965-89b08ca34aaf-kube-api-access-czm6q\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.939989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.939959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/02e511cb-2cec-4ee1-9965-89b08ca34aaf-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.941731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.941705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02e511cb-2cec-4ee1-9965-89b08ca34aaf-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.941949 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.941934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/02e511cb-2cec-4ee1-9965-89b08ca34aaf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.941991 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.941961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02e511cb-2cec-4ee1-9965-89b08ca34aaf-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:58.948722 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:58.948694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czm6q\" (UniqueName: \"kubernetes.io/projected/02e511cb-2cec-4ee1-9965-89b08ca34aaf-kube-api-access-czm6q\") pod \"opendatahub-operator-controller-manager-74d6d868c8-6d22v\" (UID: \"02e511cb-2cec-4ee1-9965-89b08ca34aaf\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:59.095999 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:59.095916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:03:59.236139 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:59.236057 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v"] Jul 19 15:03:59.238295 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:03:59.238237 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e511cb_2cec_4ee1_9965_89b08ca34aaf.slice/crio-02ce11b1a5f560a3f6671209d8b433a7f0ac53be7acfb1e308dd4aef8f519031 WatchSource:0}: Error finding container 02ce11b1a5f560a3f6671209d8b433a7f0ac53be7acfb1e308dd4aef8f519031: Status 404 returned error can't find the container with id 02ce11b1a5f560a3f6671209d8b433a7f0ac53be7acfb1e308dd4aef8f519031 Jul 19 15:03:59.374764 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:03:59.374730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" event={"ID":"02e511cb-2cec-4ee1-9965-89b08ca34aaf","Type":"ContainerStarted","Data":"02ce11b1a5f560a3f6671209d8b433a7f0ac53be7acfb1e308dd4aef8f519031"} Jul 19 15:04:00.339736 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:00.339703 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-9lxl5" Jul 19 15:04:02.393879 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:02.393837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" event={"ID":"02e511cb-2cec-4ee1-9965-89b08ca34aaf","Type":"ContainerStarted","Data":"c05947b66f2a0c3ed07d1dd3f09761517b7872f98e1813ece16369c88bb3bac0"} Jul 19 15:04:03.399332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:03.399296 2579 generic.go:356] "Generic (PLEG): container finished" podID="02e511cb-2cec-4ee1-9965-89b08ca34aaf" containerID="c05947b66f2a0c3ed07d1dd3f09761517b7872f98e1813ece16369c88bb3bac0" exitCode=0 Jul 19 15:04:03.399750 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:03.399384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" event={"ID":"02e511cb-2cec-4ee1-9965-89b08ca34aaf","Type":"ContainerDied","Data":"c05947b66f2a0c3ed07d1dd3f09761517b7872f98e1813ece16369c88bb3bac0"} Jul 19 15:04:04.405826 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:04.405793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" event={"ID":"02e511cb-2cec-4ee1-9965-89b08ca34aaf","Type":"ContainerStarted","Data":"106475a6b28f292099099766c3dd60c613845e521d5d542ea801c600789d0fd9"} Jul 19 15:04:04.406185 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:04.405880 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:04:04.430905 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:04.430844 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" podStartSLOduration=2.089336687 podStartE2EDuration="6.430823933s" totalImagesPullingTime="3.32512081s" totalInitContainerRuntime="1.016366436s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:04.428367051 +0000 UTC m=+366.288923039" watchObservedRunningTime="2026-07-19 15:04:04.430823933 +0000 UTC m=+366.291379922" Jul 19 15:04:15.413428 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:15.413396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-6d22v" Jul 19 15:04:18.441035 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.440999 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-647d897b75-mzvnn"] Jul 19 15:04:18.540036 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.540000 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-647d897b75-mzvnn"] Jul 19 15:04:18.540183 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.540094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.542894 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.542871 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 19 15:04:18.542894 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.542885 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ntjhh\"" Jul 19 15:04:18.543121 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.542873 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 19 15:04:18.544228 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.544207 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 19 15:04:18.544353 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.544230 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 19 15:04:18.629707 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.629669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e64a97a-4be8-4958-894a-28097b5d23d2-tmp\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.629942 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.629918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbjm\" (UniqueName: \"kubernetes.io/projected/1e64a97a-4be8-4958-894a-28097b5d23d2-kube-api-access-xhbjm\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.630040 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.629986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e64a97a-4be8-4958-894a-28097b5d23d2-tls-certs\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.730854 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.730767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e64a97a-4be8-4958-894a-28097b5d23d2-tmp\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.730854 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.730811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbjm\" (UniqueName: \"kubernetes.io/projected/1e64a97a-4be8-4958-894a-28097b5d23d2-kube-api-access-xhbjm\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.731075 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.730921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e64a97a-4be8-4958-894a-28097b5d23d2-tls-certs\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.733172 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.733141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e64a97a-4be8-4958-894a-28097b5d23d2-tmp\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.733332 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.733310 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e64a97a-4be8-4958-894a-28097b5d23d2-tls-certs\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.739941 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.739915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbjm\" (UniqueName: \"kubernetes.io/projected/1e64a97a-4be8-4958-894a-28097b5d23d2-kube-api-access-xhbjm\") pod \"kube-auth-proxy-647d897b75-mzvnn\" (UID: \"1e64a97a-4be8-4958-894a-28097b5d23d2\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:18.850277 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:18.850219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" Jul 19 15:04:19.029145 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:19.029121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-647d897b75-mzvnn"] Jul 19 15:04:19.031507 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:04:19.031479 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e64a97a_4be8_4958_894a_28097b5d23d2.slice/crio-99e312d4027c26cb2bde5221dcc86f4859b18bdff0ce250f0c5fd5d8810ff0df WatchSource:0}: Error finding container 99e312d4027c26cb2bde5221dcc86f4859b18bdff0ce250f0c5fd5d8810ff0df: Status 404 returned error can't find the container with id 99e312d4027c26cb2bde5221dcc86f4859b18bdff0ce250f0c5fd5d8810ff0df Jul 19 15:04:19.476683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:19.476634 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" event={"ID":"1e64a97a-4be8-4958-894a-28097b5d23d2","Type":"ContainerStarted","Data":"99e312d4027c26cb2bde5221dcc86f4859b18bdff0ce250f0c5fd5d8810ff0df"} Jul 19 15:04:22.494682 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:22.494645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" event={"ID":"1e64a97a-4be8-4958-894a-28097b5d23d2","Type":"ContainerStarted","Data":"b2a318aff55893e3cf9c84a92570857a2a0f4aff34dd447e8e6a14b5d734253b"} Jul 19 15:04:22.511583 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:04:22.511538 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-647d897b75-mzvnn" podStartSLOduration=2.126469844 podStartE2EDuration="4.511522719s" totalImagesPullingTime="2.385052875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:22.509287925 +0000 UTC m=+384.369843915" watchObservedRunningTime="2026-07-19 15:04:22.511522719 +0000 UTC m=+384.372078706" Jul 19 15:05:40.337864 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.337790 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5578bc4bdc-g8bkg"] Jul 19 15:05:40.341125 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.341108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.352102 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.352073 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5578bc4bdc-g8bkg"] Jul 19 15:05:40.460624 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-serving-cert\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.460624 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-oauth-config\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.460846 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-trusted-ca-bundle\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.460846 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-service-ca\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.460846 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-oauth-serving-cert\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.460846 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rk77\" (UniqueName: \"kubernetes.io/projected/169c8c15-208c-4617-b1e5-95cb56ac0f2e-kube-api-access-8rk77\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.461017 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.460868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-config\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561648 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-oauth-serving-cert\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561648 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rk77\" (UniqueName: \"kubernetes.io/projected/169c8c15-208c-4617-b1e5-95cb56ac0f2e-kube-api-access-8rk77\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-config\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-serving-cert\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-oauth-config\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-trusted-ca-bundle\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.561971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.561838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-service-ca\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.562575 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.562547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-oauth-serving-cert\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.562695 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.562594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-config\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.562695 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.562614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-service-ca\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.562807 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.562768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/169c8c15-208c-4617-b1e5-95cb56ac0f2e-trusted-ca-bundle\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.564197 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.564171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-oauth-config\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.564308 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.564277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/169c8c15-208c-4617-b1e5-95cb56ac0f2e-console-serving-cert\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.569725 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.569704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rk77\" (UniqueName: \"kubernetes.io/projected/169c8c15-208c-4617-b1e5-95cb56ac0f2e-kube-api-access-8rk77\") pod \"console-5578bc4bdc-g8bkg\" (UID: \"169c8c15-208c-4617-b1e5-95cb56ac0f2e\") " pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.650045 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.650021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:40.778569 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.778547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5578bc4bdc-g8bkg"] Jul 19 15:05:40.781042 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:05:40.781016 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169c8c15_208c_4617_b1e5_95cb56ac0f2e.slice/crio-09a971b18438e86fd1f9d7e0f2b08aad9da760ca996f34a4d44cb96832e64b4f WatchSource:0}: Error finding container 09a971b18438e86fd1f9d7e0f2b08aad9da760ca996f34a4d44cb96832e64b4f: Status 404 returned error can't find the container with id 09a971b18438e86fd1f9d7e0f2b08aad9da760ca996f34a4d44cb96832e64b4f Jul 19 15:05:40.839438 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:40.839410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578bc4bdc-g8bkg" event={"ID":"169c8c15-208c-4617-b1e5-95cb56ac0f2e","Type":"ContainerStarted","Data":"09a971b18438e86fd1f9d7e0f2b08aad9da760ca996f34a4d44cb96832e64b4f"} Jul 19 15:05:41.845620 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:41.845581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578bc4bdc-g8bkg" event={"ID":"169c8c15-208c-4617-b1e5-95cb56ac0f2e","Type":"ContainerStarted","Data":"304739a5eed31b2bf1c312e769975f80e9462670b768f969302b512a655812ce"} Jul 19 15:05:41.863030 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:41.862985 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5578bc4bdc-g8bkg" podStartSLOduration=1.862968202 podStartE2EDuration="1.862968202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:05:41.862646511 +0000 UTC m=+463.723202498" watchObservedRunningTime="2026-07-19 15:05:41.862968202 +0000 UTC m=+463.723524189" Jul 19 15:05:50.650394 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:50.650348 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:50.650805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:50.650413 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:50.655504 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:50.655477 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:50.892740 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:50.892708 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5578bc4bdc-g8bkg" Jul 19 15:05:50.938373 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:05:50.937103 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765f765db9-4p95c"] Jul 19 15:06:15.962299 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:15.962241 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-765f765db9-4p95c" podUID="05644a33-7bec-4bd2-9a59-044bf33a510f" containerName="console" containerID="cri-o://ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942" gracePeriod=15 Jul 19 15:06:16.207824 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.207801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:06:16.279080 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.278989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-oauth-config\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279080 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279026 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-serving-cert\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279080 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96vzq\" (UniqueName: \"kubernetes.io/projected/05644a33-7bec-4bd2-9a59-044bf33a510f-kube-api-access-96vzq\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279433 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279091 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-trusted-ca-bundle\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279433 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279124 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-console-config\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279433 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279295 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-service-ca\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279433 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-oauth-serving-cert\") pod \"05644a33-7bec-4bd2-9a59-044bf33a510f\" (UID: \"05644a33-7bec-4bd2-9a59-044bf33a510f\") " Jul 19 15:06:16.279645 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-console-config" (OuterVolumeSpecName: "console-config") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:06:16.279645 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279571 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:06:16.279745 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-service-ca" (OuterVolumeSpecName: "service-ca") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:06:16.279840 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:06:16.279840 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279832 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:16.279981 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279858 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-console-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:16.279981 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.279873 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-service-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:16.281322 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.281290 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05644a33-7bec-4bd2-9a59-044bf33a510f-kube-api-access-96vzq" (OuterVolumeSpecName: "kube-api-access-96vzq") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "kube-api-access-96vzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:06:16.281416 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.281345 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:06:16.281416 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.281364 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05644a33-7bec-4bd2-9a59-044bf33a510f" (UID: "05644a33-7bec-4bd2-9a59-044bf33a510f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:06:16.380329 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.380296 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-oauth-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:16.380329 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.380323 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05644a33-7bec-4bd2-9a59-044bf33a510f-console-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:16.380329 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.380332 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vzq\" (UniqueName: \"kubernetes.io/projected/05644a33-7bec-4bd2-9a59-044bf33a510f-kube-api-access-96vzq\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:16.380538 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:16.380343 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05644a33-7bec-4bd2-9a59-044bf33a510f-oauth-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:17.005555 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.005523 2579 generic.go:356] "Generic (PLEG): container finished" podID="05644a33-7bec-4bd2-9a59-044bf33a510f" containerID="ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942" exitCode=0 Jul 19 15:06:17.006002 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.005585 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765f765db9-4p95c" Jul 19 15:06:17.006002 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.005612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f765db9-4p95c" event={"ID":"05644a33-7bec-4bd2-9a59-044bf33a510f","Type":"ContainerDied","Data":"ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942"} Jul 19 15:06:17.006002 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.005680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765f765db9-4p95c" event={"ID":"05644a33-7bec-4bd2-9a59-044bf33a510f","Type":"ContainerDied","Data":"f23a57dfb81e273daf5480d8583d64a847316914ae1f186e96afb48557ab5f8b"} Jul 19 15:06:17.006002 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.005701 2579 scope.go:117] "RemoveContainer" containerID="ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942" Jul 19 15:06:17.014170 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.014152 2579 scope.go:117] "RemoveContainer" containerID="ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942" Jul 19 15:06:17.014580 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:17.014550 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942\": container with ID starting with ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942 not found: ID does not exist" containerID="ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942" Jul 19 15:06:17.014677 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.014590 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942"} err="failed to get container status \"ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942\": rpc error: code = NotFound desc = could not find container \"ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942\": container with ID starting with ccdddd7aacbe31150559322223baa29e2d7af6d705f632608b2b6437fa310942 not found: ID does not exist" Jul 19 15:06:17.033436 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.033404 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765f765db9-4p95c"] Jul 19 15:06:17.036879 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:17.036855 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765f765db9-4p95c"] Jul 19 15:06:18.612269 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:18.612228 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05644a33-7bec-4bd2-9a59-044bf33a510f" path="/var/lib/kubelet/pods/05644a33-7bec-4bd2-9a59-044bf33a510f/volumes" Jul 19 15:06:21.353532 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.353497 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg"] Jul 19 15:06:21.353912 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.353897 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05644a33-7bec-4bd2-9a59-044bf33a510f" containerName="console" Jul 19 15:06:21.353958 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.353914 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05644a33-7bec-4bd2-9a59-044bf33a510f" containerName="console" Jul 19 15:06:21.353992 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.353969 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05644a33-7bec-4bd2-9a59-044bf33a510f" containerName="console" Jul 19 15:06:21.356848 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.356830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.359801 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.359759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qwxx2\"" Jul 19 15:06:21.359942 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.359837 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 15:06:21.359942 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.359885 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 15:06:21.376398 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.376370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg"] Jul 19 15:06:21.427121 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.427090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b713b07-ad51-4b29-90b9-3a2813234632-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.427294 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.427153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/6b713b07-ad51-4b29-90b9-3a2813234632-kube-api-access-bh6cb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.528322 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.528283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b713b07-ad51-4b29-90b9-3a2813234632-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.528469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.528354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/6b713b07-ad51-4b29-90b9-3a2813234632-kube-api-access-bh6cb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.528645 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.528625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b713b07-ad51-4b29-90b9-3a2813234632-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.542436 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.542416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/6b713b07-ad51-4b29-90b9-3a2813234632-kube-api-access-bh6cb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.667375 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.667345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:21.803659 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:21.803635 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg"] Jul 19 15:06:21.806074 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:06:21.806048 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b713b07_ad51_4b29_90b9_3a2813234632.slice/crio-96a83232d979d9cf179f3e235a1bd3b1d28e3f28d598547c94f697cb5591a320 WatchSource:0}: Error finding container 96a83232d979d9cf179f3e235a1bd3b1d28e3f28d598547c94f697cb5591a320: Status 404 returned error can't find the container with id 96a83232d979d9cf179f3e235a1bd3b1d28e3f28d598547c94f697cb5591a320 Jul 19 15:06:22.005546 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.005460 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg"] Jul 19 15:06:22.010159 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.010128 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg"] Jul 19 15:06:22.038436 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.035647 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz"] Jul 19 15:06:22.041720 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.041691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.062638 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.062607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz"] Jul 19 15:06:22.136039 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.136006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4htpz\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.136223 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.136048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g794n\" (UniqueName: \"kubernetes.io/projected/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-kube-api-access-g794n\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4htpz\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.236909 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.236875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4htpz\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.237079 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.236918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g794n\" (UniqueName: \"kubernetes.io/projected/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-kube-api-access-g794n\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4htpz\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.237324 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.237297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4htpz\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.246424 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.246392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g794n\" (UniqueName: \"kubernetes.io/projected/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-kube-api-access-g794n\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4htpz\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.355181 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.355107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:22.744983 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:22.744959 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz"] Jul 19 15:06:22.748225 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:06:22.748198 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a62f4e1_ced1_4127_8f73_28d03f1e1c31.slice/crio-03f1c4213844ccf2029570a378aa27f80f409ccc44fcec113ab0155108f52b15 WatchSource:0}: Error finding container 03f1c4213844ccf2029570a378aa27f80f409ccc44fcec113ab0155108f52b15: Status 404 returned error can't find the container with id 03f1c4213844ccf2029570a378aa27f80f409ccc44fcec113ab0155108f52b15 Jul 19 15:06:23.048446 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:23.048346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" event={"ID":"4a62f4e1-ced1-4127-8f73-28d03f1e1c31","Type":"ContainerStarted","Data":"03f1c4213844ccf2029570a378aa27f80f409ccc44fcec113ab0155108f52b15"} Jul 19 15:06:27.073457 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.073416 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" containerName="manager" containerID="cri-o://84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a" gracePeriod=2 Jul 19 15:06:27.074971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.074946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" event={"ID":"4a62f4e1-ced1-4127-8f73-28d03f1e1c31","Type":"ContainerStarted","Data":"d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f"} Jul 19 15:06:27.075103 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.074994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:27.096622 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.096552 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" podStartSLOduration=1.386122391 podStartE2EDuration="5.096535609s" totalImagesPullingTime="3.710413218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:27.094959704 +0000 UTC m=+508.955515693" watchObservedRunningTime="2026-07-19 15:06:27.096535609 +0000 UTC m=+508.957091597" Jul 19 15:06:27.305877 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.305854 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:27.308036 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:27.308008 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:27.489792 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.489758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b713b07-ad51-4b29-90b9-3a2813234632-extensions-socket-volume\") pod \"6b713b07-ad51-4b29-90b9-3a2813234632\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " Jul 19 15:06:27.489792 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.489799 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/6b713b07-ad51-4b29-90b9-3a2813234632-kube-api-access-bh6cb\") pod \"6b713b07-ad51-4b29-90b9-3a2813234632\" (UID: \"6b713b07-ad51-4b29-90b9-3a2813234632\") " Jul 19 15:06:27.490089 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.490064 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b713b07-ad51-4b29-90b9-3a2813234632-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6b713b07-ad51-4b29-90b9-3a2813234632" (UID: "6b713b07-ad51-4b29-90b9-3a2813234632"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:06:27.491941 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.491918 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b713b07-ad51-4b29-90b9-3a2813234632-kube-api-access-bh6cb" (OuterVolumeSpecName: "kube-api-access-bh6cb") pod "6b713b07-ad51-4b29-90b9-3a2813234632" (UID: "6b713b07-ad51-4b29-90b9-3a2813234632"). InnerVolumeSpecName "kube-api-access-bh6cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:06:27.590622 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.590574 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b713b07-ad51-4b29-90b9-3a2813234632-extensions-socket-volume\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:27.590622 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:27.590616 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh6cb\" (UniqueName: \"kubernetes.io/projected/6b713b07-ad51-4b29-90b9-3a2813234632-kube-api-access-bh6cb\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:28.080588 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:28.080551 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b713b07-ad51-4b29-90b9-3a2813234632" containerID="84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a" exitCode=2 Jul 19 15:06:28.080984 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:28.080598 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:28.080984 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:28.080651 2579 scope.go:117] "RemoveContainer" containerID="84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a" Jul 19 15:06:28.082991 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:28.082962 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:28.094661 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:28.094643 2579 scope.go:117] "RemoveContainer" containerID="84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a" Jul 19 15:06:28.094849 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:28.094819 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:28.094946 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:28.094907 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a\": container with ID starting with 84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a not found: ID does not exist" containerID="84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a" Jul 19 15:06:28.094996 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:28.094943 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a"} err="failed to get container status \"84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a\": rpc error: code = NotFound desc = could not find container \"84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a\": container with ID starting with 84b3df56b9dbe7506c84545856c5aa1177812c9f1cadebdb319ffd05fe67c06a not found: ID does not exist" Jul 19 15:06:28.612162 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:28.612126 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-vhvdg\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vhvdg" Jul 19 15:06:28.613608 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:28.613586 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" path="/var/lib/kubelet/pods/6b713b07-ad51-4b29-90b9-3a2813234632/volumes" Jul 19 15:06:38.083067 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.083032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:38.517812 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.517775 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h"] Jul 19 15:06:38.518141 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.518129 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" containerName="manager" Jul 19 15:06:38.518205 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.518143 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" containerName="manager" Jul 19 15:06:38.518242 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.518219 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b713b07-ad51-4b29-90b9-3a2813234632" containerName="manager" Jul 19 15:06:38.522591 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.522574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.538327 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.538301 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h"] Jul 19 15:06:38.583346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.583305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pjf\" (UniqueName: \"kubernetes.io/projected/4eac75bc-fe02-4421-abbb-9e951d4f2753-kube-api-access-z6pjf\") pod \"kuadrant-operator-controller-manager-86878c769f-ftw8h\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.583346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.583354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eac75bc-fe02-4421-abbb-9e951d4f2753-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ftw8h\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.684102 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.684070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pjf\" (UniqueName: \"kubernetes.io/projected/4eac75bc-fe02-4421-abbb-9e951d4f2753-kube-api-access-z6pjf\") pod \"kuadrant-operator-controller-manager-86878c769f-ftw8h\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.684293 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.684104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eac75bc-fe02-4421-abbb-9e951d4f2753-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ftw8h\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.684483 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.684466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eac75bc-fe02-4421-abbb-9e951d4f2753-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ftw8h\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.692968 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.692945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pjf\" (UniqueName: \"kubernetes.io/projected/4eac75bc-fe02-4421-abbb-9e951d4f2753-kube-api-access-z6pjf\") pod \"kuadrant-operator-controller-manager-86878c769f-ftw8h\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.831986 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.831951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:38.968314 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:38.968284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h"] Jul 19 15:06:38.970657 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:06:38.970629 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eac75bc_fe02_4421_abbb_9e951d4f2753.slice/crio-45a4c940e91011b817f20cf4579c4790eebfdf04adb2d2203f5eb8580f8ed824 WatchSource:0}: Error finding container 45a4c940e91011b817f20cf4579c4790eebfdf04adb2d2203f5eb8580f8ed824: Status 404 returned error can't find the container with id 45a4c940e91011b817f20cf4579c4790eebfdf04adb2d2203f5eb8580f8ed824 Jul 19 15:06:39.133351 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:39.133318 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" event={"ID":"4eac75bc-fe02-4421-abbb-9e951d4f2753","Type":"ContainerStarted","Data":"0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d"} Jul 19 15:06:39.133735 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:39.133368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" event={"ID":"4eac75bc-fe02-4421-abbb-9e951d4f2753","Type":"ContainerStarted","Data":"45a4c940e91011b817f20cf4579c4790eebfdf04adb2d2203f5eb8580f8ed824"} Jul 19 15:06:39.133735 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:39.133448 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:39.152921 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:39.152870 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" podStartSLOduration=1.152850021 podStartE2EDuration="1.152850021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:39.151633894 +0000 UTC m=+521.012189883" watchObservedRunningTime="2026-07-19 15:06:39.152850021 +0000 UTC m=+521.013406318" Jul 19 15:06:50.141073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.141042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:06:50.185113 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.185026 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz"] Jul 19 15:06:50.185350 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.185279 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" podUID="4a62f4e1-ced1-4127-8f73-28d03f1e1c31" containerName="manager" containerID="cri-o://d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f" gracePeriod=10 Jul 19 15:06:50.432168 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.432143 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:50.487517 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.487490 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-extensions-socket-volume\") pod \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " Jul 19 15:06:50.487681 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.487540 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g794n\" (UniqueName: \"kubernetes.io/projected/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-kube-api-access-g794n\") pod \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\" (UID: \"4a62f4e1-ced1-4127-8f73-28d03f1e1c31\") " Jul 19 15:06:50.487949 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.487924 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4a62f4e1-ced1-4127-8f73-28d03f1e1c31" (UID: "4a62f4e1-ced1-4127-8f73-28d03f1e1c31"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:06:50.489612 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.489581 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-kube-api-access-g794n" (OuterVolumeSpecName: "kube-api-access-g794n") pod "4a62f4e1-ced1-4127-8f73-28d03f1e1c31" (UID: "4a62f4e1-ced1-4127-8f73-28d03f1e1c31"). InnerVolumeSpecName "kube-api-access-g794n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:06:50.588760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.588722 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-extensions-socket-volume\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:50.588760 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:50.588753 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g794n\" (UniqueName: \"kubernetes.io/projected/4a62f4e1-ced1-4127-8f73-28d03f1e1c31-kube-api-access-g794n\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:06:51.194051 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.194018 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a62f4e1-ced1-4127-8f73-28d03f1e1c31" containerID="d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f" exitCode=0 Jul 19 15:06:51.194513 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.194057 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" Jul 19 15:06:51.194513 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.194087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" event={"ID":"4a62f4e1-ced1-4127-8f73-28d03f1e1c31","Type":"ContainerDied","Data":"d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f"} Jul 19 15:06:51.194513 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.194141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz" event={"ID":"4a62f4e1-ced1-4127-8f73-28d03f1e1c31","Type":"ContainerDied","Data":"03f1c4213844ccf2029570a378aa27f80f409ccc44fcec113ab0155108f52b15"} Jul 19 15:06:51.194513 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.194173 2579 scope.go:117] "RemoveContainer" containerID="d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f" Jul 19 15:06:51.202644 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.202629 2579 scope.go:117] "RemoveContainer" containerID="d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f" Jul 19 15:06:51.202888 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:06:51.202871 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f\": container with ID starting with d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f not found: ID does not exist" containerID="d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f" Jul 19 15:06:51.202940 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.202897 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f"} err="failed to get container status \"d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f\": rpc error: code = NotFound desc = could not find container \"d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f\": container with ID starting with d96a3b06d180c65ee4f689feb18e361254cddd93a57b4afabc4133bedafba38f not found: ID does not exist" Jul 19 15:06:51.214028 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.214005 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz"] Jul 19 15:06:51.219865 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:51.219841 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4htpz"] Jul 19 15:06:52.612202 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:06:52.612170 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a62f4e1-ced1-4127-8f73-28d03f1e1c31" path="/var/lib/kubelet/pods/4a62f4e1-ced1-4127-8f73-28d03f1e1c31/volumes" Jul 19 15:07:12.653919 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.653887 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kckns"] Jul 19 15:07:12.654317 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.654229 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a62f4e1-ced1-4127-8f73-28d03f1e1c31" containerName="manager" Jul 19 15:07:12.654317 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.654241 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a62f4e1-ced1-4127-8f73-28d03f1e1c31" containerName="manager" Jul 19 15:07:12.654407 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.654336 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a62f4e1-ced1-4127-8f73-28d03f1e1c31" containerName="manager" Jul 19 15:07:12.658832 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.658811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:12.661526 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.661499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-msf75\"" Jul 19 15:07:12.663186 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.663163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kckns"] Jul 19 15:07:12.785097 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.785068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdn8\" (UniqueName: \"kubernetes.io/projected/fb790068-fc2a-4b82-93ff-0039d5e965c2-kube-api-access-jkdn8\") pod \"authorino-58cd955675-kckns\" (UID: \"fb790068-fc2a-4b82-93ff-0039d5e965c2\") " pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:12.885975 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.885944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdn8\" (UniqueName: \"kubernetes.io/projected/fb790068-fc2a-4b82-93ff-0039d5e965c2-kube-api-access-jkdn8\") pod \"authorino-58cd955675-kckns\" (UID: \"fb790068-fc2a-4b82-93ff-0039d5e965c2\") " pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:12.893872 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.893846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdn8\" (UniqueName: \"kubernetes.io/projected/fb790068-fc2a-4b82-93ff-0039d5e965c2-kube-api-access-jkdn8\") pod \"authorino-58cd955675-kckns\" (UID: \"fb790068-fc2a-4b82-93ff-0039d5e965c2\") " pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:12.970088 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:12.970005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:13.097649 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:13.097555 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kckns"] Jul 19 15:07:13.100391 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:07:13.100363 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb790068_fc2a_4b82_93ff_0039d5e965c2.slice/crio-4a56596dbcfc51ed59c40fbf8beab216af4f62c08bc4807d7c43291aaa847ee2 WatchSource:0}: Error finding container 4a56596dbcfc51ed59c40fbf8beab216af4f62c08bc4807d7c43291aaa847ee2: Status 404 returned error can't find the container with id 4a56596dbcfc51ed59c40fbf8beab216af4f62c08bc4807d7c43291aaa847ee2 Jul 19 15:07:13.295429 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:13.295344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kckns" event={"ID":"fb790068-fc2a-4b82-93ff-0039d5e965c2","Type":"ContainerStarted","Data":"4a56596dbcfc51ed59c40fbf8beab216af4f62c08bc4807d7c43291aaa847ee2"} Jul 19 15:07:16.314898 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:16.314869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kckns" event={"ID":"fb790068-fc2a-4b82-93ff-0039d5e965c2","Type":"ContainerStarted","Data":"9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765"} Jul 19 15:07:16.329024 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:16.328977 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kckns" podStartSLOduration=1.181028151 podStartE2EDuration="4.328961622s" totalImagesPullingTime="3.147933471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:16.328193489 +0000 UTC m=+558.188749485" watchObservedRunningTime="2026-07-19 15:07:16.328961622 +0000 UTC m=+558.189517643" Jul 19 15:07:17.795380 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:17.795351 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kckns"] Jul 19 15:07:18.325637 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:18.325598 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kckns" podUID="fb790068-fc2a-4b82-93ff-0039d5e965c2" containerName="authorino" containerID="cri-o://9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765" gracePeriod=30 Jul 19 15:07:18.569544 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:18.569522 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:18.739529 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:18.739494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkdn8\" (UniqueName: \"kubernetes.io/projected/fb790068-fc2a-4b82-93ff-0039d5e965c2-kube-api-access-jkdn8\") pod \"fb790068-fc2a-4b82-93ff-0039d5e965c2\" (UID: \"fb790068-fc2a-4b82-93ff-0039d5e965c2\") " Jul 19 15:07:18.741565 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:18.741540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb790068-fc2a-4b82-93ff-0039d5e965c2-kube-api-access-jkdn8" (OuterVolumeSpecName: "kube-api-access-jkdn8") pod "fb790068-fc2a-4b82-93ff-0039d5e965c2" (UID: "fb790068-fc2a-4b82-93ff-0039d5e965c2"). InnerVolumeSpecName "kube-api-access-jkdn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:07:18.840663 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:18.840634 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdn8\" (UniqueName: \"kubernetes.io/projected/fb790068-fc2a-4b82-93ff-0039d5e965c2-kube-api-access-jkdn8\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:07:19.331544 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.331511 2579 generic.go:356] "Generic (PLEG): container finished" podID="fb790068-fc2a-4b82-93ff-0039d5e965c2" containerID="9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765" exitCode=0 Jul 19 15:07:19.331724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.331556 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kckns" Jul 19 15:07:19.331724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.331604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kckns" event={"ID":"fb790068-fc2a-4b82-93ff-0039d5e965c2","Type":"ContainerDied","Data":"9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765"} Jul 19 15:07:19.331724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.331648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kckns" event={"ID":"fb790068-fc2a-4b82-93ff-0039d5e965c2","Type":"ContainerDied","Data":"4a56596dbcfc51ed59c40fbf8beab216af4f62c08bc4807d7c43291aaa847ee2"} Jul 19 15:07:19.331724 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.331668 2579 scope.go:117] "RemoveContainer" containerID="9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765" Jul 19 15:07:19.339994 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.339971 2579 scope.go:117] "RemoveContainer" containerID="9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765" Jul 19 15:07:19.340249 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:07:19.340233 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765\": container with ID starting with 9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765 not found: ID does not exist" containerID="9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765" Jul 19 15:07:19.340330 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.340280 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765"} err="failed to get container status \"9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765\": rpc error: code = NotFound desc = could not find container \"9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765\": container with ID starting with 9423270b39d4c9ccde6ddd4073894b727a8d15cd53a74490b477e77a93e66765 not found: ID does not exist" Jul 19 15:07:19.352693 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.352672 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kckns"] Jul 19 15:07:19.355990 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:19.355964 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kckns"] Jul 19 15:07:20.612758 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:20.612724 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb790068-fc2a-4b82-93ff-0039d5e965c2" path="/var/lib/kubelet/pods/fb790068-fc2a-4b82-93ff-0039d5e965c2/volumes" Jul 19 15:07:40.428892 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.428859 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8c25l"] Jul 19 15:07:40.429373 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.429197 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb790068-fc2a-4b82-93ff-0039d5e965c2" containerName="authorino" Jul 19 15:07:40.429373 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.429207 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb790068-fc2a-4b82-93ff-0039d5e965c2" containerName="authorino" Jul 19 15:07:40.429373 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.429297 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb790068-fc2a-4b82-93ff-0039d5e965c2" containerName="authorino" Jul 19 15:07:40.432443 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.432423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:40.435141 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.435113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-msf75\"" Jul 19 15:07:40.437043 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.437022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8c25l"] Jul 19 15:07:40.535983 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.535952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cl95\" (UniqueName: \"kubernetes.io/projected/48207b9a-663a-492f-8692-3fc959c533eb-kube-api-access-9cl95\") pod \"authorino-5d4b67c446-8c25l\" (UID: \"48207b9a-663a-492f-8692-3fc959c533eb\") " pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:40.636781 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.636745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cl95\" (UniqueName: \"kubernetes.io/projected/48207b9a-663a-492f-8692-3fc959c533eb-kube-api-access-9cl95\") pod \"authorino-5d4b67c446-8c25l\" (UID: \"48207b9a-663a-492f-8692-3fc959c533eb\") " pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:40.646097 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.646069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cl95\" (UniqueName: \"kubernetes.io/projected/48207b9a-663a-492f-8692-3fc959c533eb-kube-api-access-9cl95\") pod \"authorino-5d4b67c446-8c25l\" (UID: \"48207b9a-663a-492f-8692-3fc959c533eb\") " pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:40.654791 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.654761 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8c25l"] Jul 19 15:07:40.655075 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.655061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:40.678690 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.678659 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ccc6797f5-r7n94"] Jul 19 15:07:40.681891 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.681838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:40.692041 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.692010 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccc6797f5-r7n94"] Jul 19 15:07:40.736956 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.736926 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ccc6797f5-r7n94"] Jul 19 15:07:40.737196 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:07:40.737176 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zgb2p], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5ccc6797f5-r7n94" podUID="332ce796-84c3-44d6-9912-7d8f843c0f8d" Jul 19 15:07:40.737266 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.737208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgb2p\" (UniqueName: \"kubernetes.io/projected/332ce796-84c3-44d6-9912-7d8f843c0f8d-kube-api-access-zgb2p\") pod \"authorino-5ccc6797f5-r7n94\" (UID: \"332ce796-84c3-44d6-9912-7d8f843c0f8d\") " pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:40.758286 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.758238 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-685b85b9f5-6ndlm"] Jul 19 15:07:40.761183 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.761159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:40.763700 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.763674 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 19 15:07:40.768942 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.768832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-685b85b9f5-6ndlm"] Jul 19 15:07:40.789526 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.789405 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8c25l"] Jul 19 15:07:40.791682 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:07:40.791654 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48207b9a_663a_492f_8692_3fc959c533eb.slice/crio-53e382147b0507769cf247a7a090068057727d65c966dc425fa375075fab50c7 WatchSource:0}: Error finding container 53e382147b0507769cf247a7a090068057727d65c966dc425fa375075fab50c7: Status 404 returned error can't find the container with id 53e382147b0507769cf247a7a090068057727d65c966dc425fa375075fab50c7 Jul 19 15:07:40.838630 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.838600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s48c\" (UniqueName: \"kubernetes.io/projected/8bc384e9-8326-41df-8279-859987cacb1c-kube-api-access-7s48c\") pod \"authorino-685b85b9f5-6ndlm\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:40.838630 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.838633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8bc384e9-8326-41df-8279-859987cacb1c-tls-cert\") pod \"authorino-685b85b9f5-6ndlm\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:40.838848 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.838742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgb2p\" (UniqueName: \"kubernetes.io/projected/332ce796-84c3-44d6-9912-7d8f843c0f8d-kube-api-access-zgb2p\") pod \"authorino-5ccc6797f5-r7n94\" (UID: \"332ce796-84c3-44d6-9912-7d8f843c0f8d\") " pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:40.847282 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.847236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgb2p\" (UniqueName: \"kubernetes.io/projected/332ce796-84c3-44d6-9912-7d8f843c0f8d-kube-api-access-zgb2p\") pod \"authorino-5ccc6797f5-r7n94\" (UID: \"332ce796-84c3-44d6-9912-7d8f843c0f8d\") " pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:40.940071 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.939928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8bc384e9-8326-41df-8279-859987cacb1c-tls-cert\") pod \"authorino-685b85b9f5-6ndlm\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:40.940209 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.940150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s48c\" (UniqueName: \"kubernetes.io/projected/8bc384e9-8326-41df-8279-859987cacb1c-kube-api-access-7s48c\") pod \"authorino-685b85b9f5-6ndlm\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:40.942214 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.942196 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8bc384e9-8326-41df-8279-859987cacb1c-tls-cert\") pod \"authorino-685b85b9f5-6ndlm\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:40.947681 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:40.947659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s48c\" (UniqueName: \"kubernetes.io/projected/8bc384e9-8326-41df-8279-859987cacb1c-kube-api-access-7s48c\") pod \"authorino-685b85b9f5-6ndlm\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:41.071893 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.071860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:07:41.202199 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.202175 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-685b85b9f5-6ndlm"] Jul 19 15:07:41.203890 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:07:41.203865 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc384e9_8326_41df_8279_859987cacb1c.slice/crio-8657bcf4fae7f797eaa7722a4ae6f01d8ee7f21f1a2a7df5d4b9c180660feb28 WatchSource:0}: Error finding container 8657bcf4fae7f797eaa7722a4ae6f01d8ee7f21f1a2a7df5d4b9c180660feb28: Status 404 returned error can't find the container with id 8657bcf4fae7f797eaa7722a4ae6f01d8ee7f21f1a2a7df5d4b9c180660feb28 Jul 19 15:07:41.433300 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.433241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" event={"ID":"8bc384e9-8326-41df-8279-859987cacb1c","Type":"ContainerStarted","Data":"8657bcf4fae7f797eaa7722a4ae6f01d8ee7f21f1a2a7df5d4b9c180660feb28"} Jul 19 15:07:41.434685 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.434652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8c25l" event={"ID":"48207b9a-663a-492f-8692-3fc959c533eb","Type":"ContainerStarted","Data":"9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507"} Jul 19 15:07:41.434805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.434707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8c25l" event={"ID":"48207b9a-663a-492f-8692-3fc959c533eb","Type":"ContainerStarted","Data":"53e382147b0507769cf247a7a090068057727d65c966dc425fa375075fab50c7"} Jul 19 15:07:41.434805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.434666 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:41.434805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.434721 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-8c25l" podUID="48207b9a-663a-492f-8692-3fc959c533eb" containerName="authorino" containerID="cri-o://9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507" gracePeriod=30 Jul 19 15:07:41.440682 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.440663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:41.448939 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.448904 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-8c25l" podStartSLOduration=1.09491114 podStartE2EDuration="1.448889451s" totalImagesPullingTime="353.978311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:41.448446363 +0000 UTC m=+583.309002351" watchObservedRunningTime="2026-07-19 15:07:41.448889451 +0000 UTC m=+583.309445439" Jul 19 15:07:41.547624 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.547599 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgb2p\" (UniqueName: \"kubernetes.io/projected/332ce796-84c3-44d6-9912-7d8f843c0f8d-kube-api-access-zgb2p\") pod \"332ce796-84c3-44d6-9912-7d8f843c0f8d\" (UID: \"332ce796-84c3-44d6-9912-7d8f843c0f8d\") " Jul 19 15:07:41.549386 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.549361 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332ce796-84c3-44d6-9912-7d8f843c0f8d-kube-api-access-zgb2p" (OuterVolumeSpecName: "kube-api-access-zgb2p") pod "332ce796-84c3-44d6-9912-7d8f843c0f8d" (UID: "332ce796-84c3-44d6-9912-7d8f843c0f8d"). InnerVolumeSpecName "kube-api-access-zgb2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:07:41.649334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.649297 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgb2p\" (UniqueName: \"kubernetes.io/projected/332ce796-84c3-44d6-9912-7d8f843c0f8d-kube-api-access-zgb2p\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:07:41.693526 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.693502 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:41.750343 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.750229 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cl95\" (UniqueName: \"kubernetes.io/projected/48207b9a-663a-492f-8692-3fc959c533eb-kube-api-access-9cl95\") pod \"48207b9a-663a-492f-8692-3fc959c533eb\" (UID: \"48207b9a-663a-492f-8692-3fc959c533eb\") " Jul 19 15:07:41.752207 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.752185 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48207b9a-663a-492f-8692-3fc959c533eb-kube-api-access-9cl95" (OuterVolumeSpecName: "kube-api-access-9cl95") pod "48207b9a-663a-492f-8692-3fc959c533eb" (UID: "48207b9a-663a-492f-8692-3fc959c533eb"). InnerVolumeSpecName "kube-api-access-9cl95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:07:41.851241 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:41.851210 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cl95\" (UniqueName: \"kubernetes.io/projected/48207b9a-663a-492f-8692-3fc959c533eb-kube-api-access-9cl95\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:07:42.441133 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.441096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" event={"ID":"8bc384e9-8326-41df-8279-859987cacb1c","Type":"ContainerStarted","Data":"8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e"} Jul 19 15:07:42.442341 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.442313 2579 generic.go:356] "Generic (PLEG): container finished" podID="48207b9a-663a-492f-8692-3fc959c533eb" containerID="9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507" exitCode=0 Jul 19 15:07:42.442341 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.442342 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8c25l" Jul 19 15:07:42.442506 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.442373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8c25l" event={"ID":"48207b9a-663a-492f-8692-3fc959c533eb","Type":"ContainerDied","Data":"9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507"} Jul 19 15:07:42.442506 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.442407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-8c25l" event={"ID":"48207b9a-663a-492f-8692-3fc959c533eb","Type":"ContainerDied","Data":"53e382147b0507769cf247a7a090068057727d65c966dc425fa375075fab50c7"} Jul 19 15:07:42.442506 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.442428 2579 scope.go:117] "RemoveContainer" containerID="9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507" Jul 19 15:07:42.442646 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.442577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccc6797f5-r7n94" Jul 19 15:07:42.451434 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.451340 2579 scope.go:117] "RemoveContainer" containerID="9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507" Jul 19 15:07:42.451642 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:07:42.451615 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507\": container with ID starting with 9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507 not found: ID does not exist" containerID="9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507" Jul 19 15:07:42.451816 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.451651 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507"} err="failed to get container status \"9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507\": rpc error: code = NotFound desc = could not find container \"9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507\": container with ID starting with 9da9a9631d80c13290ee4ba6ed8429250480772f903a8c7636eebb464980a507 not found: ID does not exist" Jul 19 15:07:42.456484 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.456447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" podStartSLOduration=2.126973347 podStartE2EDuration="2.456435161s" totalImagesPullingTime="329.461814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:42.455756888 +0000 UTC m=+584.316312877" watchObservedRunningTime="2026-07-19 15:07:42.456435161 +0000 UTC m=+584.316991149" Jul 19 15:07:42.469463 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.469442 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8c25l"] Jul 19 15:07:42.474346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.474324 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8c25l"] Jul 19 15:07:42.499880 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.499849 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ccc6797f5-r7n94"] Jul 19 15:07:42.502883 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.502857 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ccc6797f5-r7n94"] Jul 19 15:07:42.614320 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.614247 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="332ce796-84c3-44d6-9912-7d8f843c0f8d" path="/var/lib/kubelet/pods/332ce796-84c3-44d6-9912-7d8f843c0f8d/volumes" Jul 19 15:07:42.614638 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:42.614619 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48207b9a-663a-492f-8692-3fc959c533eb" path="/var/lib/kubelet/pods/48207b9a-663a-492f-8692-3fc959c533eb/volumes" Jul 19 15:07:58.531095 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:58.531058 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:07:58.531830 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:07:58.531807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:08:23.678250 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.678173 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-skwkp"] Jul 19 15:08:23.680491 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.678645 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48207b9a-663a-492f-8692-3fc959c533eb" containerName="authorino" Jul 19 15:08:23.680491 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.678664 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="48207b9a-663a-492f-8692-3fc959c533eb" containerName="authorino" Jul 19 15:08:23.680491 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.678729 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="48207b9a-663a-492f-8692-3fc959c533eb" containerName="authorino" Jul 19 15:08:23.681352 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.681335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.683825 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.683804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 19 15:08:23.683940 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.683804 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xkmfd\"" Jul 19 15:08:23.690039 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.690010 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-skwkp"] Jul 19 15:08:23.735581 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.735546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b984f288-7f6d-469c-9fb1-1eadbf63cc2c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-skwkp\" (UID: \"b984f288-7f6d-469c-9fb1-1eadbf63cc2c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.735581 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.735580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmd2\" (UniqueName: \"kubernetes.io/projected/b984f288-7f6d-469c-9fb1-1eadbf63cc2c-kube-api-access-nsmd2\") pod \"payload-pre-processing-7677b774bb-skwkp\" (UID: \"b984f288-7f6d-469c-9fb1-1eadbf63cc2c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.836659 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.836618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b984f288-7f6d-469c-9fb1-1eadbf63cc2c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-skwkp\" (UID: \"b984f288-7f6d-469c-9fb1-1eadbf63cc2c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.836823 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.836668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmd2\" (UniqueName: \"kubernetes.io/projected/b984f288-7f6d-469c-9fb1-1eadbf63cc2c-kube-api-access-nsmd2\") pod \"payload-pre-processing-7677b774bb-skwkp\" (UID: \"b984f288-7f6d-469c-9fb1-1eadbf63cc2c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.837434 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.837409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b984f288-7f6d-469c-9fb1-1eadbf63cc2c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-skwkp\" (UID: \"b984f288-7f6d-469c-9fb1-1eadbf63cc2c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.845810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.845785 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmd2\" (UniqueName: \"kubernetes.io/projected/b984f288-7f6d-469c-9fb1-1eadbf63cc2c-kube-api-access-nsmd2\") pod \"payload-pre-processing-7677b774bb-skwkp\" (UID: \"b984f288-7f6d-469c-9fb1-1eadbf63cc2c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:23.992913 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:23.992828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:24.126657 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:24.126627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-skwkp"] Jul 19 15:08:24.129902 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:08:24.129873 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb984f288_7f6d_469c_9fb1_1eadbf63cc2c.slice/crio-baf70d4f43d7eb67d1ea4716beeac98f63e37f0999d49cada6fa342b21cfd381 WatchSource:0}: Error finding container baf70d4f43d7eb67d1ea4716beeac98f63e37f0999d49cada6fa342b21cfd381: Status 404 returned error can't find the container with id baf70d4f43d7eb67d1ea4716beeac98f63e37f0999d49cada6fa342b21cfd381 Jul 19 15:08:24.131780 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:24.131763 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:08:24.644217 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:24.644134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" event={"ID":"b984f288-7f6d-469c-9fb1-1eadbf63cc2c","Type":"ContainerStarted","Data":"baf70d4f43d7eb67d1ea4716beeac98f63e37f0999d49cada6fa342b21cfd381"} Jul 19 15:08:26.655140 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:26.655105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" event={"ID":"b984f288-7f6d-469c-9fb1-1eadbf63cc2c","Type":"ContainerStarted","Data":"62bd8fed07c603024e1dc4d35f77c5eea125aebbe381cf5d462472c6d416d5a0"} Jul 19 15:08:26.655572 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:26.655235 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:26.673409 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:26.673360 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" podStartSLOduration=2.169746536 podStartE2EDuration="3.673346233s" totalImagesPullingTime="1.503599697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:08:26.670726967 +0000 UTC m=+628.531282957" watchObservedRunningTime="2026-07-19 15:08:26.673346233 +0000 UTC m=+628.533902221" Jul 19 15:08:37.662908 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:37.662876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-skwkp" Jul 19 15:08:55.401698 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.401659 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76"] Jul 19 15:08:55.405673 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.405651 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.409496 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.409469 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 19 15:08:55.409617 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.409530 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 19 15:08:55.409617 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.409551 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c7xpt\"" Jul 19 15:08:55.409749 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.409472 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 19 15:08:55.414636 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.414614 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76"] Jul 19 15:08:55.426780 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.426751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.426878 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.426793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.426878 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.426812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.426878 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.426851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.426991 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.426908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.426991 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.426934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92t8\" (UniqueName: \"kubernetes.io/projected/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-kube-api-access-f92t8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528017 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.527976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528017 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528292 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528292 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528292 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92t8\" (UniqueName: \"kubernetes.io/projected/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-kube-api-access-f92t8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528573 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.528669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.528649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.530372 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.530344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.530512 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.530494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.535971 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.535949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92t8\" (UniqueName: \"kubernetes.io/projected/a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021-kube-api-access-f92t8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fkn76\" (UID: \"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.718463 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.718378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:08:55.862063 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:55.862031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76"] Jul 19 15:08:55.864054 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:08:55.864023 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68a3c4a_b0b4_4a8f_8e7f_86aa03df5021.slice/crio-6e4e19e4045c15e56cb95dda206a907afadc8bd782cbf13e8b55f45b868429f9 WatchSource:0}: Error finding container 6e4e19e4045c15e56cb95dda206a907afadc8bd782cbf13e8b55f45b868429f9: Status 404 returned error can't find the container with id 6e4e19e4045c15e56cb95dda206a907afadc8bd782cbf13e8b55f45b868429f9 Jul 19 15:08:56.798521 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:56.798458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" event={"ID":"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021","Type":"ContainerStarted","Data":"6e4e19e4045c15e56cb95dda206a907afadc8bd782cbf13e8b55f45b868429f9"} Jul 19 15:08:57.804802 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:57.804760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" event={"ID":"a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021","Type":"ContainerStarted","Data":"67c41de82a6128449fc4c87613075bd772c92109b23347a0e28615755f20ae2d"} Jul 19 15:08:57.821980 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:08:57.821923 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" podStartSLOduration=1.68002692 podStartE2EDuration="2.821908759s" totalImagesPullingTime="1.141881839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:08:57.820835105 +0000 UTC m=+659.681391106" watchObservedRunningTime="2026-07-19 15:08:57.821908759 +0000 UTC m=+659.682464748" Jul 19 15:09:05.719509 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:05.719464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:09:05.719930 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:05.719549 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:09:05.732113 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:05.732085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:09:05.853502 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:05.853471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fkn76" Jul 19 15:09:15.998105 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:15.998058 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9"] Jul 19 15:09:16.001837 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.001815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.004556 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.004538 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 19 15:09:16.011634 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.011608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9"] Jul 19 15:09:16.121705 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.121668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.121705 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.121709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.121935 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.121789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.121935 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.121865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wb44\" (UniqueName: \"kubernetes.io/projected/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-kube-api-access-9wb44\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.121935 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.121886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.122062 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.121948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223089 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb44\" (UniqueName: \"kubernetes.io/projected/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-kube-api-access-9wb44\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223089 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223399 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223399 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223399 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223399 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223581 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223646 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.223701 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.223683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.225554 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.225529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.225803 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.225782 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.231578 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.231559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb44\" (UniqueName: \"kubernetes.io/projected/6e0ffa46-206e-454c-a8ed-b2f6f4c375b2-kube-api-access-9wb44\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nlrp9\" (UID: \"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.313686 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.313603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:16.450941 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.450916 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9"] Jul 19 15:09:16.452937 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:09:16.452909 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0ffa46_206e_454c_a8ed_b2f6f4c375b2.slice/crio-9634c154c5ea82fb1d71d9038e74df22d6376f6f7af1ef3eac14d5c5a385888f WatchSource:0}: Error finding container 9634c154c5ea82fb1d71d9038e74df22d6376f6f7af1ef3eac14d5c5a385888f: Status 404 returned error can't find the container with id 9634c154c5ea82fb1d71d9038e74df22d6376f6f7af1ef3eac14d5c5a385888f Jul 19 15:09:16.895562 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.895482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" event={"ID":"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2","Type":"ContainerStarted","Data":"34ee1ab12b7068bee9b4fff1849e607148ef4bc1f8c276a63e35571ac5ce8028"} Jul 19 15:09:16.895562 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.895540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" event={"ID":"6e0ffa46-206e-454c-a8ed-b2f6f4c375b2","Type":"ContainerStarted","Data":"9634c154c5ea82fb1d71d9038e74df22d6376f6f7af1ef3eac14d5c5a385888f"} Jul 19 15:09:16.913856 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:16.913803 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" podStartSLOduration=1.737118154 podStartE2EDuration="1.913787871s" totalImagesPullingTime="176.669717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:09:16.911074459 +0000 UTC m=+678.771630447" watchObservedRunningTime="2026-07-19 15:09:16.913787871 +0000 UTC m=+678.774343858" Jul 19 15:09:26.314534 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:26.314499 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:26.315049 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:26.314558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:26.327237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:26.327212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:26.957154 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:26.957126 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nlrp9" Jul 19 15:09:42.502997 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.502961 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v"] Jul 19 15:09:42.510951 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.509231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.514078 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.513862 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 19 15:09:42.528279 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.520700 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v"] Jul 19 15:09:42.572353 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.572314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.572528 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.572358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.572528 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.572390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.572528 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.572467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.572528 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.572503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ck7\" (UniqueName: \"kubernetes.io/projected/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-kube-api-access-d8ck7\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.572528 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.572527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.673987 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.673954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.673987 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.673995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674229 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674229 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674229 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ck7\" (UniqueName: \"kubernetes.io/projected/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-kube-api-access-d8ck7\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674229 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674494 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674554 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.674632 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.674610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.676440 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.676406 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.676652 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.676631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.681692 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.681668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ck7\" (UniqueName: \"kubernetes.io/projected/6f7c7987-4a0e-4b4c-8d74-26280b5ced00-kube-api-access-d8ck7\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pz97v\" (UID: \"6f7c7987-4a0e-4b4c-8d74-26280b5ced00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.845458 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.845376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:42.997733 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:42.997702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v"] Jul 19 15:09:42.999089 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:09:42.999052 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f7c7987_4a0e_4b4c_8d74_26280b5ced00.slice/crio-8cc6d7471ba932efb728de20f4ac4b4be95726e8ccef1dbf7703b829ed0812db WatchSource:0}: Error finding container 8cc6d7471ba932efb728de20f4ac4b4be95726e8ccef1dbf7703b829ed0812db: Status 404 returned error can't find the container with id 8cc6d7471ba932efb728de20f4ac4b4be95726e8ccef1dbf7703b829ed0812db Jul 19 15:09:43.021399 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:43.021368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" event={"ID":"6f7c7987-4a0e-4b4c-8d74-26280b5ced00","Type":"ContainerStarted","Data":"8cc6d7471ba932efb728de20f4ac4b4be95726e8ccef1dbf7703b829ed0812db"} Jul 19 15:09:44.027472 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:44.027432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" event={"ID":"6f7c7987-4a0e-4b4c-8d74-26280b5ced00","Type":"ContainerStarted","Data":"aab0d510b3ffea865a9827176fbc5714d10fc9d0a44a9932fef56bdbc91462c3"} Jul 19 15:09:44.044929 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:44.044878 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" podStartSLOduration=1.8593763490000002 podStartE2EDuration="2.044861184s" totalImagesPullingTime="185.484835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:09:44.042740303 +0000 UTC m=+705.903296292" watchObservedRunningTime="2026-07-19 15:09:44.044861184 +0000 UTC m=+705.905417173" Jul 19 15:09:52.846048 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:52.845998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:52.846492 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:52.846060 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:52.858427 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:52.858399 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:09:53.093411 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:09:53.093380 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pz97v" Jul 19 15:10:52.133066 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.133032 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8866d997d-pqkt2"] Jul 19 15:10:52.136929 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.136908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.144893 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.144816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8866d997d-pqkt2"] Jul 19 15:10:52.229818 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.229784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r552\" (UniqueName: \"kubernetes.io/projected/20e71070-3d5c-420e-8ecb-f400155ff385-kube-api-access-7r552\") pod \"authorino-8866d997d-pqkt2\" (UID: \"20e71070-3d5c-420e-8ecb-f400155ff385\") " pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.229996 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.229831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20e71070-3d5c-420e-8ecb-f400155ff385-tls-cert\") pod \"authorino-8866d997d-pqkt2\" (UID: \"20e71070-3d5c-420e-8ecb-f400155ff385\") " pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.330738 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.330645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r552\" (UniqueName: \"kubernetes.io/projected/20e71070-3d5c-420e-8ecb-f400155ff385-kube-api-access-7r552\") pod \"authorino-8866d997d-pqkt2\" (UID: \"20e71070-3d5c-420e-8ecb-f400155ff385\") " pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.330738 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.330695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20e71070-3d5c-420e-8ecb-f400155ff385-tls-cert\") pod \"authorino-8866d997d-pqkt2\" (UID: \"20e71070-3d5c-420e-8ecb-f400155ff385\") " pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.333272 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.333232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20e71070-3d5c-420e-8ecb-f400155ff385-tls-cert\") pod \"authorino-8866d997d-pqkt2\" (UID: \"20e71070-3d5c-420e-8ecb-f400155ff385\") " pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.339096 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.339065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r552\" (UniqueName: \"kubernetes.io/projected/20e71070-3d5c-420e-8ecb-f400155ff385-kube-api-access-7r552\") pod \"authorino-8866d997d-pqkt2\" (UID: \"20e71070-3d5c-420e-8ecb-f400155ff385\") " pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.448580 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.448548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8866d997d-pqkt2" Jul 19 15:10:52.590282 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:52.590234 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8866d997d-pqkt2"] Jul 19 15:10:52.592556 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:10:52.592529 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20e71070_3d5c_420e_8ecb_f400155ff385.slice/crio-484c15d9fa63a2232d067321039252bac720ac869813fcdd5329fb29e8515e49 WatchSource:0}: Error finding container 484c15d9fa63a2232d067321039252bac720ac869813fcdd5329fb29e8515e49: Status 404 returned error can't find the container with id 484c15d9fa63a2232d067321039252bac720ac869813fcdd5329fb29e8515e49 Jul 19 15:10:53.376902 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.376865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8866d997d-pqkt2" event={"ID":"20e71070-3d5c-420e-8ecb-f400155ff385","Type":"ContainerStarted","Data":"93d62a464d028367e6f16f5f68635c0d7bd9bea516193244b6050bd7f778fdad"} Jul 19 15:10:53.377281 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.376916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8866d997d-pqkt2" event={"ID":"20e71070-3d5c-420e-8ecb-f400155ff385","Type":"ContainerStarted","Data":"484c15d9fa63a2232d067321039252bac720ac869813fcdd5329fb29e8515e49"} Jul 19 15:10:53.397944 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.397874 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8866d997d-pqkt2" podStartSLOduration=1.01247577 podStartE2EDuration="1.397850968s" totalImagesPullingTime="385.375198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:10:53.39511031 +0000 UTC m=+775.255666299" watchObservedRunningTime="2026-07-19 15:10:53.397850968 +0000 UTC m=+775.258406958" Jul 19 15:10:53.429757 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.429722 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-685b85b9f5-6ndlm"] Jul 19 15:10:53.430180 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.430152 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" podUID="8bc384e9-8326-41df-8279-859987cacb1c" containerName="authorino" containerID="cri-o://8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e" gracePeriod=30 Jul 19 15:10:53.698711 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.698685 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:10:53.748605 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.748567 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s48c\" (UniqueName: \"kubernetes.io/projected/8bc384e9-8326-41df-8279-859987cacb1c-kube-api-access-7s48c\") pod \"8bc384e9-8326-41df-8279-859987cacb1c\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " Jul 19 15:10:53.750050 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.750029 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8bc384e9-8326-41df-8279-859987cacb1c-tls-cert\") pod \"8bc384e9-8326-41df-8279-859987cacb1c\" (UID: \"8bc384e9-8326-41df-8279-859987cacb1c\") " Jul 19 15:10:53.754009 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.753975 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc384e9-8326-41df-8279-859987cacb1c-kube-api-access-7s48c" (OuterVolumeSpecName: "kube-api-access-7s48c") pod "8bc384e9-8326-41df-8279-859987cacb1c" (UID: "8bc384e9-8326-41df-8279-859987cacb1c"). InnerVolumeSpecName "kube-api-access-7s48c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:10:53.764077 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.764051 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc384e9-8326-41df-8279-859987cacb1c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8bc384e9-8326-41df-8279-859987cacb1c" (UID: "8bc384e9-8326-41df-8279-859987cacb1c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:10:53.851037 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.851001 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8bc384e9-8326-41df-8279-859987cacb1c-tls-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:10:53.851037 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:53.851030 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s48c\" (UniqueName: \"kubernetes.io/projected/8bc384e9-8326-41df-8279-859987cacb1c-kube-api-access-7s48c\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:10:54.383447 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.383410 2579 generic.go:356] "Generic (PLEG): container finished" podID="8bc384e9-8326-41df-8279-859987cacb1c" containerID="8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e" exitCode=0 Jul 19 15:10:54.383844 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.383464 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" Jul 19 15:10:54.383844 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.383487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" event={"ID":"8bc384e9-8326-41df-8279-859987cacb1c","Type":"ContainerDied","Data":"8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e"} Jul 19 15:10:54.383844 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.383528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-685b85b9f5-6ndlm" event={"ID":"8bc384e9-8326-41df-8279-859987cacb1c","Type":"ContainerDied","Data":"8657bcf4fae7f797eaa7722a4ae6f01d8ee7f21f1a2a7df5d4b9c180660feb28"} Jul 19 15:10:54.383844 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.383544 2579 scope.go:117] "RemoveContainer" containerID="8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e" Jul 19 15:10:54.394448 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.394392 2579 scope.go:117] "RemoveContainer" containerID="8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e" Jul 19 15:10:54.395093 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:10:54.395065 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e\": container with ID starting with 8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e not found: ID does not exist" containerID="8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e" Jul 19 15:10:54.395195 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.395105 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e"} err="failed to get container status \"8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e\": rpc error: code = NotFound desc = could not find container \"8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e\": container with ID starting with 8fb0f4165ef1e44b3541dcf1bf92621504ea0ab50dd622d1744feee31475343e not found: ID does not exist" Jul 19 15:10:54.409061 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.409035 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-685b85b9f5-6ndlm"] Jul 19 15:10:54.421164 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.421127 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-685b85b9f5-6ndlm"] Jul 19 15:10:54.612985 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:10:54.612886 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc384e9-8326-41df-8279-859987cacb1c" path="/var/lib/kubelet/pods/8bc384e9-8326-41df-8279-859987cacb1c/volumes" Jul 19 15:12:58.562896 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:12:58.562867 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:12:58.563739 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:12:58.563723 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:17:50.704252 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:50.704159 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h"] Jul 19 15:17:50.704847 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:50.704424 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" podUID="4eac75bc-fe02-4421-abbb-9e951d4f2753" containerName="manager" containerID="cri-o://0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d" gracePeriod=10 Jul 19 15:17:54.447138 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.447112 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:17:54.483078 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.483040 2579 generic.go:356] "Generic (PLEG): container finished" podID="4eac75bc-fe02-4421-abbb-9e951d4f2753" containerID="0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d" exitCode=0 Jul 19 15:17:54.483250 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.483083 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" Jul 19 15:17:54.483250 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.483093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" event={"ID":"4eac75bc-fe02-4421-abbb-9e951d4f2753","Type":"ContainerDied","Data":"0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d"} Jul 19 15:17:54.483250 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.483148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h" event={"ID":"4eac75bc-fe02-4421-abbb-9e951d4f2753","Type":"ContainerDied","Data":"45a4c940e91011b817f20cf4579c4790eebfdf04adb2d2203f5eb8580f8ed824"} Jul 19 15:17:54.483250 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.483180 2579 scope.go:117] "RemoveContainer" containerID="0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d" Jul 19 15:17:54.494552 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.494529 2579 scope.go:117] "RemoveContainer" containerID="0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d" Jul 19 15:17:54.494881 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:17:54.494852 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d\": container with ID starting with 0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d not found: ID does not exist" containerID="0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d" Jul 19 15:17:54.495000 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.494897 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d"} err="failed to get container status \"0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d\": rpc error: code = NotFound desc = could not find container \"0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d\": container with ID starting with 0042c342475e26446194b0648a65d892b929fb0386ca0c3e218e7423f10fce1d not found: ID does not exist" Jul 19 15:17:54.551907 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.551870 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eac75bc-fe02-4421-abbb-9e951d4f2753-extensions-socket-volume\") pod \"4eac75bc-fe02-4421-abbb-9e951d4f2753\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " Jul 19 15:17:54.552080 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.551995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6pjf\" (UniqueName: \"kubernetes.io/projected/4eac75bc-fe02-4421-abbb-9e951d4f2753-kube-api-access-z6pjf\") pod \"4eac75bc-fe02-4421-abbb-9e951d4f2753\" (UID: \"4eac75bc-fe02-4421-abbb-9e951d4f2753\") " Jul 19 15:17:54.552336 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.552308 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eac75bc-fe02-4421-abbb-9e951d4f2753-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4eac75bc-fe02-4421-abbb-9e951d4f2753" (UID: "4eac75bc-fe02-4421-abbb-9e951d4f2753"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:17:54.554195 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.554164 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eac75bc-fe02-4421-abbb-9e951d4f2753-kube-api-access-z6pjf" (OuterVolumeSpecName: "kube-api-access-z6pjf") pod "4eac75bc-fe02-4421-abbb-9e951d4f2753" (UID: "4eac75bc-fe02-4421-abbb-9e951d4f2753"). InnerVolumeSpecName "kube-api-access-z6pjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:17:54.652853 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.652821 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eac75bc-fe02-4421-abbb-9e951d4f2753-extensions-socket-volume\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:17:54.652853 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.652849 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pjf\" (UniqueName: \"kubernetes.io/projected/4eac75bc-fe02-4421-abbb-9e951d4f2753-kube-api-access-z6pjf\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:17:54.799727 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.799697 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h"] Jul 19 15:17:54.804377 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:54.804349 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ftw8h"] Jul 19 15:17:56.612789 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:56.612760 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eac75bc-fe02-4421-abbb-9e951d4f2753" path="/var/lib/kubelet/pods/4eac75bc-fe02-4421-abbb-9e951d4f2753/volumes" Jul 19 15:17:58.599245 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:58.599212 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:17:58.600945 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:17:58.600921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:18:51.855052 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.854973 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62"] Jul 19 15:18:51.855550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.855357 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eac75bc-fe02-4421-abbb-9e951d4f2753" containerName="manager" Jul 19 15:18:51.855550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.855369 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eac75bc-fe02-4421-abbb-9e951d4f2753" containerName="manager" Jul 19 15:18:51.855550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.855388 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc384e9-8326-41df-8279-859987cacb1c" containerName="authorino" Jul 19 15:18:51.855550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.855394 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc384e9-8326-41df-8279-859987cacb1c" containerName="authorino" Jul 19 15:18:51.855550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.855454 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eac75bc-fe02-4421-abbb-9e951d4f2753" containerName="manager" Jul 19 15:18:51.855550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.855466 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc384e9-8326-41df-8279-859987cacb1c" containerName="authorino" Jul 19 15:18:51.858546 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.858530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:51.863665 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.863644 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qwxx2\"" Jul 19 15:18:51.867479 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.867455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48jt\" (UniqueName: \"kubernetes.io/projected/cb408c64-5593-4628-8576-60e02f5770c4-kube-api-access-k48jt\") pod \"kuadrant-operator-controller-manager-86878c769f-jkl62\" (UID: \"cb408c64-5593-4628-8576-60e02f5770c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:51.867588 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.867522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb408c64-5593-4628-8576-60e02f5770c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jkl62\" (UID: \"cb408c64-5593-4628-8576-60e02f5770c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:51.874453 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.874434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62"] Jul 19 15:18:51.968582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.968544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48jt\" (UniqueName: \"kubernetes.io/projected/cb408c64-5593-4628-8576-60e02f5770c4-kube-api-access-k48jt\") pod \"kuadrant-operator-controller-manager-86878c769f-jkl62\" (UID: \"cb408c64-5593-4628-8576-60e02f5770c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:51.968751 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.968606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb408c64-5593-4628-8576-60e02f5770c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jkl62\" (UID: \"cb408c64-5593-4628-8576-60e02f5770c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:51.968963 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.968947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb408c64-5593-4628-8576-60e02f5770c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jkl62\" (UID: \"cb408c64-5593-4628-8576-60e02f5770c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:51.976957 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:51.976932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48jt\" (UniqueName: \"kubernetes.io/projected/cb408c64-5593-4628-8576-60e02f5770c4-kube-api-access-k48jt\") pod \"kuadrant-operator-controller-manager-86878c769f-jkl62\" (UID: \"cb408c64-5593-4628-8576-60e02f5770c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:52.168876 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:52.168845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:52.308802 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:52.308779 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62"] Jul 19 15:18:52.310990 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:18:52.310960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb408c64_5593_4628_8576_60e02f5770c4.slice/crio-251e630cc1a15b5feaf186ce78293a65c5bf74df22b58981795696c52ea3383e WatchSource:0}: Error finding container 251e630cc1a15b5feaf186ce78293a65c5bf74df22b58981795696c52ea3383e: Status 404 returned error can't find the container with id 251e630cc1a15b5feaf186ce78293a65c5bf74df22b58981795696c52ea3383e Jul 19 15:18:52.760128 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:52.760090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" event={"ID":"cb408c64-5593-4628-8576-60e02f5770c4","Type":"ContainerStarted","Data":"347faa4c39436e80b06e73e5777a2f6e7554e92dbaa44b2593d91b959f13a640"} Jul 19 15:18:52.760320 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:52.760144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" event={"ID":"cb408c64-5593-4628-8576-60e02f5770c4","Type":"ContainerStarted","Data":"251e630cc1a15b5feaf186ce78293a65c5bf74df22b58981795696c52ea3383e"} Jul 19 15:18:52.760320 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:52.760183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:18:52.783061 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:18:52.783011 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" podStartSLOduration=1.782994483 podStartE2EDuration="1.782994483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:18:52.780868136 +0000 UTC m=+1254.641424125" watchObservedRunningTime="2026-07-19 15:18:52.782994483 +0000 UTC m=+1254.643550471" Jul 19 15:19:03.767346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:19:03.767317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkl62" Jul 19 15:22:58.641908 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:22:58.641878 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:22:58.644370 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:22:58.644348 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:27:58.672216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:27:58.672187 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:27:58.676019 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:27:58.675997 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:32:16.198415 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.198373 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp"] Jul 19 15:32:16.201960 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.201942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.208058 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.207953 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cc1f323c\"/\"default-dockercfg-5rvrx\"" Jul 19 15:32:16.208226 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.208154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cc1f323c\"/\"e2e-lifecycle-model-cc1f323c-kserve-self-signed-certs\"" Jul 19 15:32:16.208398 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.208375 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cc1f323c\"/\"kube-root-ca.crt\"" Jul 19 15:32:16.208568 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.208389 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cc1f323c\"/\"openshift-service-ca.crt\"" Jul 19 15:32:16.210912 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.210872 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp"] Jul 19 15:32:16.228565 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.228535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tls-certs\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.228719 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.228581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-home\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.228719 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.228654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgkh\" (UniqueName: \"kubernetes.io/projected/2130a2b9-bf8d-4b70-94cf-64410e0953b9-kube-api-access-mzgkh\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.228815 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.228736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-model-cache\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.228815 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.228762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-dshm\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.228891 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.228820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tmp-dir\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330136 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tmp-dir\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tls-certs\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-home\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkh\" (UniqueName: \"kubernetes.io/projected/2130a2b9-bf8d-4b70-94cf-64410e0953b9-kube-api-access-mzgkh\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-model-cache\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330357 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-dshm\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330640 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tmp-dir\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330640 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-home\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.330727 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.330695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-model-cache\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.332461 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.332432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-dshm\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.332742 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.332723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tls-certs\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.338441 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.338419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgkh\" (UniqueName: \"kubernetes.io/projected/2130a2b9-bf8d-4b70-94cf-64410e0953b9-kube-api-access-mzgkh\") pod \"e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.512984 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.512886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:16.645753 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.645686 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp"] Jul 19 15:32:16.648498 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:32:16.648472 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2130a2b9_bf8d_4b70_94cf_64410e0953b9.slice/crio-882e8f90d27d81af2bff75bd1df94e4bbe0af9d938781636cfe5ed6fb8d7b4b0 WatchSource:0}: Error finding container 882e8f90d27d81af2bff75bd1df94e4bbe0af9d938781636cfe5ed6fb8d7b4b0: Status 404 returned error can't find the container with id 882e8f90d27d81af2bff75bd1df94e4bbe0af9d938781636cfe5ed6fb8d7b4b0 Jul 19 15:32:16.761102 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.761054 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" event={"ID":"2130a2b9-bf8d-4b70-94cf-64410e0953b9","Type":"ContainerStarted","Data":"b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9"} Jul 19 15:32:16.761294 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.761114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" event={"ID":"2130a2b9-bf8d-4b70-94cf-64410e0953b9","Type":"ContainerStarted","Data":"882e8f90d27d81af2bff75bd1df94e4bbe0af9d938781636cfe5ed6fb8d7b4b0"} Jul 19 15:32:16.779031 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:16.778940 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" podStartSLOduration=0.778923586 podStartE2EDuration="778.923586ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:32:16.776468142 +0000 UTC m=+2058.637024143" watchObservedRunningTime="2026-07-19 15:32:16.778923586 +0000 UTC m=+2058.639479573" Jul 19 15:32:26.513883 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:26.513796 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:26.514287 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:26.513886 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:26.526784 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:26.526756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:26.818348 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:26.818252 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:32:58.701873 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:58.701846 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:32:58.707632 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:32:58.707610 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:33:14.006384 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:14.006350 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp"] Jul 19 15:33:29.028553 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.028490 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" podUID="2130a2b9-bf8d-4b70-94cf-64410e0953b9" containerName="main" containerID="cri-o://b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9" gracePeriod=45 Jul 19 15:33:29.285618 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.285557 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:33:29.322572 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322536 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tls-certs\") pod \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " Jul 19 15:33:29.322572 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322575 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzgkh\" (UniqueName: \"kubernetes.io/projected/2130a2b9-bf8d-4b70-94cf-64410e0953b9-kube-api-access-mzgkh\") pod \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " Jul 19 15:33:29.322803 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-dshm\") pod \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " Jul 19 15:33:29.322803 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-model-cache\") pod \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " Jul 19 15:33:29.322803 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tmp-dir\") pod \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " Jul 19 15:33:29.322803 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322728 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-home\") pod \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\" (UID: \"2130a2b9-bf8d-4b70-94cf-64410e0953b9\") " Jul 19 15:33:29.323016 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322970 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "2130a2b9-bf8d-4b70-94cf-64410e0953b9" (UID: "2130a2b9-bf8d-4b70-94cf-64410e0953b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:29.323016 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.322986 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2130a2b9-bf8d-4b70-94cf-64410e0953b9" (UID: "2130a2b9-bf8d-4b70-94cf-64410e0953b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:29.323157 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.323135 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-home" (OuterVolumeSpecName: "home") pod "2130a2b9-bf8d-4b70-94cf-64410e0953b9" (UID: "2130a2b9-bf8d-4b70-94cf-64410e0953b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:29.325042 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.325019 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2130a2b9-bf8d-4b70-94cf-64410e0953b9" (UID: "2130a2b9-bf8d-4b70-94cf-64410e0953b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:33:29.325220 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.325205 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-dshm" (OuterVolumeSpecName: "dshm") pod "2130a2b9-bf8d-4b70-94cf-64410e0953b9" (UID: "2130a2b9-bf8d-4b70-94cf-64410e0953b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:29.325310 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.325251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2130a2b9-bf8d-4b70-94cf-64410e0953b9-kube-api-access-mzgkh" (OuterVolumeSpecName: "kube-api-access-mzgkh") pod "2130a2b9-bf8d-4b70-94cf-64410e0953b9" (UID: "2130a2b9-bf8d-4b70-94cf-64410e0953b9"). InnerVolumeSpecName "kube-api-access-mzgkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:33:29.423523 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.423489 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:33:29.423694 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.423540 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:33:29.423694 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.423550 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:33:29.423694 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.423560 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2130a2b9-bf8d-4b70-94cf-64410e0953b9-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:33:29.423694 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.423568 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzgkh\" (UniqueName: \"kubernetes.io/projected/2130a2b9-bf8d-4b70-94cf-64410e0953b9-kube-api-access-mzgkh\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:33:29.423694 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:29.423576 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2130a2b9-bf8d-4b70-94cf-64410e0953b9-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:33:30.111148 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.111115 2579 generic.go:356] "Generic (PLEG): container finished" podID="2130a2b9-bf8d-4b70-94cf-64410e0953b9" containerID="b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9" exitCode=0 Jul 19 15:33:30.111628 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.111156 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" Jul 19 15:33:30.111628 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.111181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" event={"ID":"2130a2b9-bf8d-4b70-94cf-64410e0953b9","Type":"ContainerDied","Data":"b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9"} Jul 19 15:33:30.111628 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.111218 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp" event={"ID":"2130a2b9-bf8d-4b70-94cf-64410e0953b9","Type":"ContainerDied","Data":"882e8f90d27d81af2bff75bd1df94e4bbe0af9d938781636cfe5ed6fb8d7b4b0"} Jul 19 15:33:30.111628 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.111234 2579 scope.go:117] "RemoveContainer" containerID="b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9" Jul 19 15:33:30.121218 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.121199 2579 scope.go:117] "RemoveContainer" containerID="b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9" Jul 19 15:33:30.121517 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:33:30.121495 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9\": container with ID starting with b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9 not found: ID does not exist" containerID="b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9" Jul 19 15:33:30.121588 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.121526 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9"} err="failed to get container status \"b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9\": rpc error: code = NotFound desc = could not find container \"b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9\": container with ID starting with b478142e3d67f4fc04123414e6013e61b202c89df85f1378a2c94a76988301d9 not found: ID does not exist" Jul 19 15:33:30.132919 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.132889 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp"] Jul 19 15:33:30.138134 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.138113 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cc1f323c/e2e-lifecycle-model-cc1f323c-kserve-9d5b89cf5-p55cp"] Jul 19 15:33:30.613312 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:33:30.613279 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2130a2b9-bf8d-4b70-94cf-64410e0953b9" path="/var/lib/kubelet/pods/2130a2b9-bf8d-4b70-94cf-64410e0953b9/volumes" Jul 19 15:35:12.797408 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.797377 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2"] Jul 19 15:35:12.797868 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.797763 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2130a2b9-bf8d-4b70-94cf-64410e0953b9" containerName="main" Jul 19 15:35:12.797868 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.797773 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2130a2b9-bf8d-4b70-94cf-64410e0953b9" containerName="main" Jul 19 15:35:12.797868 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.797832 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2130a2b9-bf8d-4b70-94cf-64410e0953b9" containerName="main" Jul 19 15:35:12.801301 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.801283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:12.804624 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.804594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1dc017\"/\"default-dockercfg-vzqwl\"" Jul 19 15:35:12.804624 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.804613 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1dc017\"/\"kube-root-ca.crt\"" Jul 19 15:35:12.805940 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.805915 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1dc017\"/\"test-model-1dc017-kserve-self-signed-certs\"" Jul 19 15:35:12.806092 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.806071 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1dc017\"/\"openshift-service-ca.crt\"" Jul 19 15:35:12.811539 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.811519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2"] Jul 19 15:35:12.922808 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.922775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f5d507-4e07-4632-83d0-40da258daca7-tls-certs\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:12.922983 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.922840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-dshm\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:12.922983 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.922902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-home\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:12.922983 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.922933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-model-cache\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:12.922983 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.922954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-tmp-dir\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:12.923125 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:12.923062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfk6\" (UniqueName: \"kubernetes.io/projected/83f5d507-4e07-4632-83d0-40da258daca7-kube-api-access-5wfk6\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.023707 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.023669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-tmp-dir\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.023886 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.023726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfk6\" (UniqueName: \"kubernetes.io/projected/83f5d507-4e07-4632-83d0-40da258daca7-kube-api-access-5wfk6\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.023886 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.023753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f5d507-4e07-4632-83d0-40da258daca7-tls-certs\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.023886 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.023796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-dshm\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.023886 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.023831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-home\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.023886 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.023853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-model-cache\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.024191 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.024161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-tmp-dir\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.024191 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.024184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-model-cache\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.024344 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.024244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-home\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.026181 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.026163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-dshm\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.026340 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.026317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f5d507-4e07-4632-83d0-40da258daca7-tls-certs\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.031815 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.031794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfk6\" (UniqueName: \"kubernetes.io/projected/83f5d507-4e07-4632-83d0-40da258daca7-kube-api-access-5wfk6\") pod \"test-model-1dc017-kserve-698f576d6f-df8g2\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.111293 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.111192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:13.245399 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.245370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2"] Jul 19 15:35:13.247796 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:35:13.247769 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83f5d507_4e07_4632_83d0_40da258daca7.slice/crio-15d1f5fde2257cab92cf7cc0a61db176415d63dad1d5c32593ccd8e4ab6e8ff9 WatchSource:0}: Error finding container 15d1f5fde2257cab92cf7cc0a61db176415d63dad1d5c32593ccd8e4ab6e8ff9: Status 404 returned error can't find the container with id 15d1f5fde2257cab92cf7cc0a61db176415d63dad1d5c32593ccd8e4ab6e8ff9 Jul 19 15:35:13.617422 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.617388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" event={"ID":"83f5d507-4e07-4632-83d0-40da258daca7","Type":"ContainerStarted","Data":"faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72"} Jul 19 15:35:13.617422 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.617426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" event={"ID":"83f5d507-4e07-4632-83d0-40da258daca7","Type":"ContainerStarted","Data":"15d1f5fde2257cab92cf7cc0a61db176415d63dad1d5c32593ccd8e4ab6e8ff9"} Jul 19 15:35:13.636193 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:13.636118 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" podStartSLOduration=1.6360978259999999 podStartE2EDuration="1.636097826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:35:13.633081998 +0000 UTC m=+2235.493637989" watchObservedRunningTime="2026-07-19 15:35:13.636097826 +0000 UTC m=+2235.496653814" Jul 19 15:35:23.111840 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:23.111744 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:23.111840 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:23.111799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:23.124275 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:23.124238 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:23.676812 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:23.676782 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:35:36.503547 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.503506 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b"] Jul 19 15:35:36.507197 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.507174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.511993 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.511919 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5e6749\"/\"openshift-service-ca.crt\"" Jul 19 15:35:36.512150 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.512102 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5e6749\"/\"test-model-5e6749-kserve-self-signed-certs\"" Jul 19 15:35:36.512214 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.512154 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5e6749\"/\"kube-root-ca.crt\"" Jul 19 15:35:36.512214 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.512178 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5e6749\"/\"default-dockercfg-c5mwr\"" Jul 19 15:35:36.516039 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.515762 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b"] Jul 19 15:35:36.556778 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.556740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-dshm\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.556778 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.556781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-home\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.557015 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.556813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tmp-dir\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.557015 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.556903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhllh\" (UniqueName: \"kubernetes.io/projected/7ab884e7-8050-4f47-b170-93c9fabcb7f8-kube-api-access-zhllh\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.557015 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.556947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tls-certs\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.557015 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.556984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-model-cache\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658173 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tmp-dir\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhllh\" (UniqueName: \"kubernetes.io/projected/7ab884e7-8050-4f47-b170-93c9fabcb7f8-kube-api-access-zhllh\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tls-certs\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-model-cache\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-dshm\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-home\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658732 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tmp-dir\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658732 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-model-cache\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.658732 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.658706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-home\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.660571 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.660546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-dshm\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.660845 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.660826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tls-certs\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.668313 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.668289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhllh\" (UniqueName: \"kubernetes.io/projected/7ab884e7-8050-4f47-b170-93c9fabcb7f8-kube-api-access-zhllh\") pod \"test-model-5e6749-kserve-6bb7fbf4f-kfb4b\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.821965 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.821880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:36.957447 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:36.957423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b"] Jul 19 15:35:36.959551 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:35:36.959524 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ab884e7_8050_4f47_b170_93c9fabcb7f8.slice/crio-f48c16c0111631e8edcb8d80bf5e6ee5148cff4e800493813f191e7970b20044 WatchSource:0}: Error finding container f48c16c0111631e8edcb8d80bf5e6ee5148cff4e800493813f191e7970b20044: Status 404 returned error can't find the container with id f48c16c0111631e8edcb8d80bf5e6ee5148cff4e800493813f191e7970b20044 Jul 19 15:35:37.734646 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:37.734592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" event={"ID":"7ab884e7-8050-4f47-b170-93c9fabcb7f8","Type":"ContainerStarted","Data":"8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27"} Jul 19 15:35:37.735133 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:37.734657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" event={"ID":"7ab884e7-8050-4f47-b170-93c9fabcb7f8","Type":"ContainerStarted","Data":"f48c16c0111631e8edcb8d80bf5e6ee5148cff4e800493813f191e7970b20044"} Jul 19 15:35:37.753541 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:37.753493 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" podStartSLOduration=1.753477618 podStartE2EDuration="1.753477618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:35:37.751166328 +0000 UTC m=+2259.611722318" watchObservedRunningTime="2026-07-19 15:35:37.753477618 +0000 UTC m=+2259.614033641" Jul 19 15:35:46.822205 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:46.822154 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:46.822205 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:46.822212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:46.836156 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:46.836129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:47.797209 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:47.797181 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:35:56.405289 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:56.405240 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2"] Jul 19 15:35:57.319966 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:35:57.319937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b"] Jul 19 15:36:11.428138 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.428094 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" podUID="83f5d507-4e07-4632-83d0-40da258daca7" containerName="main" containerID="cri-o://faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72" gracePeriod=45 Jul 19 15:36:11.691377 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.691314 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:36:11.800358 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800326 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-tmp-dir\") pod \"83f5d507-4e07-4632-83d0-40da258daca7\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " Jul 19 15:36:11.800567 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800377 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-home\") pod \"83f5d507-4e07-4632-83d0-40da258daca7\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " Jul 19 15:36:11.800567 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wfk6\" (UniqueName: \"kubernetes.io/projected/83f5d507-4e07-4632-83d0-40da258daca7-kube-api-access-5wfk6\") pod \"83f5d507-4e07-4632-83d0-40da258daca7\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " Jul 19 15:36:11.800567 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-model-cache\") pod \"83f5d507-4e07-4632-83d0-40da258daca7\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " Jul 19 15:36:11.800567 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800491 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f5d507-4e07-4632-83d0-40da258daca7-tls-certs\") pod \"83f5d507-4e07-4632-83d0-40da258daca7\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " Jul 19 15:36:11.800567 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-dshm\") pod \"83f5d507-4e07-4632-83d0-40da258daca7\" (UID: \"83f5d507-4e07-4632-83d0-40da258daca7\") " Jul 19 15:36:11.800796 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83f5d507-4e07-4632-83d0-40da258daca7" (UID: "83f5d507-4e07-4632-83d0-40da258daca7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:11.800849 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-model-cache" (OuterVolumeSpecName: "model-cache") pod "83f5d507-4e07-4632-83d0-40da258daca7" (UID: "83f5d507-4e07-4632-83d0-40da258daca7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:11.800960 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800939 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:11.800960 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.800961 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:11.801147 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.801039 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-home" (OuterVolumeSpecName: "home") pod "83f5d507-4e07-4632-83d0-40da258daca7" (UID: "83f5d507-4e07-4632-83d0-40da258daca7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:11.802854 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.802828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f5d507-4e07-4632-83d0-40da258daca7-kube-api-access-5wfk6" (OuterVolumeSpecName: "kube-api-access-5wfk6") pod "83f5d507-4e07-4632-83d0-40da258daca7" (UID: "83f5d507-4e07-4632-83d0-40da258daca7"). InnerVolumeSpecName "kube-api-access-5wfk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:11.803099 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.803078 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f5d507-4e07-4632-83d0-40da258daca7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83f5d507-4e07-4632-83d0-40da258daca7" (UID: "83f5d507-4e07-4632-83d0-40da258daca7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:36:11.803164 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.803106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-dshm" (OuterVolumeSpecName: "dshm") pod "83f5d507-4e07-4632-83d0-40da258daca7" (UID: "83f5d507-4e07-4632-83d0-40da258daca7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:11.901740 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.901696 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:11.901740 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.901736 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfk6\" (UniqueName: \"kubernetes.io/projected/83f5d507-4e07-4632-83d0-40da258daca7-kube-api-access-5wfk6\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:11.901740 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.901747 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f5d507-4e07-4632-83d0-40da258daca7-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:11.901740 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.901757 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f5d507-4e07-4632-83d0-40da258daca7-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:11.911732 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.911697 2579 generic.go:356] "Generic (PLEG): container finished" podID="83f5d507-4e07-4632-83d0-40da258daca7" containerID="faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72" exitCode=0 Jul 19 15:36:11.911880 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.911743 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" Jul 19 15:36:11.911880 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.911741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" event={"ID":"83f5d507-4e07-4632-83d0-40da258daca7","Type":"ContainerDied","Data":"faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72"} Jul 19 15:36:11.911880 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.911867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2" event={"ID":"83f5d507-4e07-4632-83d0-40da258daca7","Type":"ContainerDied","Data":"15d1f5fde2257cab92cf7cc0a61db176415d63dad1d5c32593ccd8e4ab6e8ff9"} Jul 19 15:36:11.912035 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.911894 2579 scope.go:117] "RemoveContainer" containerID="faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72" Jul 19 15:36:11.921186 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.921163 2579 scope.go:117] "RemoveContainer" containerID="faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72" Jul 19 15:36:11.921500 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:36:11.921480 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72\": container with ID starting with faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72 not found: ID does not exist" containerID="faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72" Jul 19 15:36:11.921563 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.921510 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72"} err="failed to get container status \"faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72\": rpc error: code = NotFound desc = could not find container \"faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72\": container with ID starting with faaa386273861ee48b87fc80a3a5ecb5759ba2b95bb43aec2481c2996e61cf72 not found: ID does not exist" Jul 19 15:36:11.933000 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.932974 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2"] Jul 19 15:36:11.939098 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:11.939077 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1dc017/test-model-1dc017-kserve-698f576d6f-df8g2"] Jul 19 15:36:12.340796 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.340756 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" podUID="7ab884e7-8050-4f47-b170-93c9fabcb7f8" containerName="main" containerID="cri-o://8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27" gracePeriod=45 Jul 19 15:36:12.596916 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.596852 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:36:12.613596 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.613566 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f5d507-4e07-4632-83d0-40da258daca7" path="/var/lib/kubelet/pods/83f5d507-4e07-4632-83d0-40da258daca7/volumes" Jul 19 15:36:12.709341 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709301 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-model-cache\") pod \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " Jul 19 15:36:12.709341 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709343 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-home\") pod \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " Jul 19 15:36:12.709598 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709383 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhllh\" (UniqueName: \"kubernetes.io/projected/7ab884e7-8050-4f47-b170-93c9fabcb7f8-kube-api-access-zhllh\") pod \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " Jul 19 15:36:12.709598 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709451 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tmp-dir\") pod \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " Jul 19 15:36:12.709598 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709491 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tls-certs\") pod \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " Jul 19 15:36:12.709598 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709585 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-home" (OuterVolumeSpecName: "home") pod "7ab884e7-8050-4f47-b170-93c9fabcb7f8" (UID: "7ab884e7-8050-4f47-b170-93c9fabcb7f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:12.709775 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709601 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ab884e7-8050-4f47-b170-93c9fabcb7f8" (UID: "7ab884e7-8050-4f47-b170-93c9fabcb7f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:12.709775 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-dshm\") pod \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\" (UID: \"7ab884e7-8050-4f47-b170-93c9fabcb7f8\") " Jul 19 15:36:12.709775 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ab884e7-8050-4f47-b170-93c9fabcb7f8" (UID: "7ab884e7-8050-4f47-b170-93c9fabcb7f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:12.709972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709945 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:12.709972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709968 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:12.709972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.709979 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:12.711692 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.711670 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ab884e7-8050-4f47-b170-93c9fabcb7f8" (UID: "7ab884e7-8050-4f47-b170-93c9fabcb7f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:36:12.711692 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.711677 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-dshm" (OuterVolumeSpecName: "dshm") pod "7ab884e7-8050-4f47-b170-93c9fabcb7f8" (UID: "7ab884e7-8050-4f47-b170-93c9fabcb7f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:12.711821 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.711676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab884e7-8050-4f47-b170-93c9fabcb7f8-kube-api-access-zhllh" (OuterVolumeSpecName: "kube-api-access-zhllh") pod "7ab884e7-8050-4f47-b170-93c9fabcb7f8" (UID: "7ab884e7-8050-4f47-b170-93c9fabcb7f8"). InnerVolumeSpecName "kube-api-access-zhllh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:12.810795 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.810754 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab884e7-8050-4f47-b170-93c9fabcb7f8-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:12.810795 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.810787 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab884e7-8050-4f47-b170-93c9fabcb7f8-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:12.810795 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.810798 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhllh\" (UniqueName: \"kubernetes.io/projected/7ab884e7-8050-4f47-b170-93c9fabcb7f8-kube-api-access-zhllh\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:36:12.918690 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.918659 2579 generic.go:356] "Generic (PLEG): container finished" podID="7ab884e7-8050-4f47-b170-93c9fabcb7f8" containerID="8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27" exitCode=0 Jul 19 15:36:12.918862 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.918696 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" Jul 19 15:36:12.918862 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.918746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" event={"ID":"7ab884e7-8050-4f47-b170-93c9fabcb7f8","Type":"ContainerDied","Data":"8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27"} Jul 19 15:36:12.918862 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.918788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b" event={"ID":"7ab884e7-8050-4f47-b170-93c9fabcb7f8","Type":"ContainerDied","Data":"f48c16c0111631e8edcb8d80bf5e6ee5148cff4e800493813f191e7970b20044"} Jul 19 15:36:12.918862 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.918811 2579 scope.go:117] "RemoveContainer" containerID="8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27" Jul 19 15:36:12.927785 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.927768 2579 scope.go:117] "RemoveContainer" containerID="8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27" Jul 19 15:36:12.928041 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:36:12.928025 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27\": container with ID starting with 8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27 not found: ID does not exist" containerID="8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27" Jul 19 15:36:12.928081 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.928053 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27"} err="failed to get container status \"8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27\": rpc error: code = NotFound desc = could not find container \"8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27\": container with ID starting with 8317e022c04c42b3643e3d42b2fc6859627973b646719d3ff657b339e6501a27 not found: ID does not exist" Jul 19 15:36:12.941129 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.941091 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b"] Jul 19 15:36:12.946435 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:12.946406 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5e6749/test-model-5e6749-kserve-6bb7fbf4f-kfb4b"] Jul 19 15:36:14.613204 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:36:14.613166 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab884e7-8050-4f47-b170-93c9fabcb7f8" path="/var/lib/kubelet/pods/7ab884e7-8050-4f47-b170-93c9fabcb7f8/volumes" Jul 19 15:37:58.733331 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:37:58.733295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:37:58.740989 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:37:58.740958 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:38:27.201317 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201230 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr"] Jul 19 15:38:27.201755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201618 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab884e7-8050-4f47-b170-93c9fabcb7f8" containerName="main" Jul 19 15:38:27.201755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201630 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab884e7-8050-4f47-b170-93c9fabcb7f8" containerName="main" Jul 19 15:38:27.201755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201652 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f5d507-4e07-4632-83d0-40da258daca7" containerName="main" Jul 19 15:38:27.201755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201657 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f5d507-4e07-4632-83d0-40da258daca7" containerName="main" Jul 19 15:38:27.201755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201736 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f5d507-4e07-4632-83d0-40da258daca7" containerName="main" Jul 19 15:38:27.201755 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.201747 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab884e7-8050-4f47-b170-93c9fabcb7f8" containerName="main" Jul 19 15:38:27.205010 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.204994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.209076 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.209049 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-51b4ec\"/\"auth-test-model-51b4ec-kserve-self-signed-certs\"" Jul 19 15:38:27.209199 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.209089 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-51b4ec\"/\"openshift-service-ca.crt\"" Jul 19 15:38:27.209199 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.209088 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-51b4ec\"/\"default-dockercfg-lwwrh\"" Jul 19 15:38:27.209199 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.209091 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-51b4ec\"/\"kube-root-ca.crt\"" Jul 19 15:38:27.213194 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.213166 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr"] Jul 19 15:38:27.269285 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.269222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc53d11-55f5-411a-8727-19eb83a231f5-tls-certs\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.269487 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.269306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgmw\" (UniqueName: \"kubernetes.io/projected/8dc53d11-55f5-411a-8727-19eb83a231f5-kube-api-access-jfgmw\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.269487 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.269354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-home\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.269487 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.269369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-tmp-dir\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.269487 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.269444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-dshm\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.269679 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.269528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-model-cache\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.370601 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.370558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc53d11-55f5-411a-8727-19eb83a231f5-tls-certs\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.370810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.370640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgmw\" (UniqueName: \"kubernetes.io/projected/8dc53d11-55f5-411a-8727-19eb83a231f5-kube-api-access-jfgmw\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.370810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.370745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-home\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.370810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.370783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-tmp-dir\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.370992 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.370830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-dshm\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.370992 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.370882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-model-cache\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.371153 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.371131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-home\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.371197 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.371153 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-tmp-dir\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.371282 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.371197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-model-cache\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.372981 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.372959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-dshm\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.373218 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.373202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc53d11-55f5-411a-8727-19eb83a231f5-tls-certs\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.378591 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.378568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgmw\" (UniqueName: \"kubernetes.io/projected/8dc53d11-55f5-411a-8727-19eb83a231f5-kube-api-access-jfgmw\") pod \"auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.515469 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.515371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:27.652414 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:27.652384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr"] Jul 19 15:38:27.654077 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:38:27.654050 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc53d11_55f5_411a_8727_19eb83a231f5.slice/crio-b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c WatchSource:0}: Error finding container b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c: Status 404 returned error can't find the container with id b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c Jul 19 15:38:28.590527 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:28.590484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" event={"ID":"8dc53d11-55f5-411a-8727-19eb83a231f5","Type":"ContainerStarted","Data":"3386c487b14894d0f7d613bf953d45906eaf1f722edbb360ed255c8453615c41"} Jul 19 15:38:28.590922 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:28.590541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" event={"ID":"8dc53d11-55f5-411a-8727-19eb83a231f5","Type":"ContainerStarted","Data":"b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c"} Jul 19 15:38:28.611579 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:28.611514 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" podStartSLOduration=1.611494333 podStartE2EDuration="1.611494333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:38:28.606687542 +0000 UTC m=+2430.467243530" watchObservedRunningTime="2026-07-19 15:38:28.611494333 +0000 UTC m=+2430.472050325" Jul 19 15:38:37.516391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:37.516339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:37.516391 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:37.516401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:37.529000 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:37.528973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:37.651039 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:38:37.651010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:38:58.506516 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:38:58.506481 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dc53d11_55f5_411a_8727_19eb83a231f5.slice/crio-b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c\": RecentStats: unable to find data in memory cache]" Jul 19 15:39:12.313667 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.313627 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x"] Jul 19 15:39:12.317175 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.317154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.320199 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.320176 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"openshift-service-ca.crt\"" Jul 19 15:39:12.321365 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.321341 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"kube-root-ca.crt\"" Jul 19 15:39:12.321457 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.321432 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"default-dockercfg-zkbc6\"" Jul 19 15:39:12.321511 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.321491 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"auth-test-model-c17947-kserve-self-signed-certs\"" Jul 19 15:39:12.326522 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.326497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x"] Jul 19 15:39:12.396129 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.396095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b382ed64-0983-460b-a89f-12716ddb46a7-tls-certs\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.396346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.396143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-model-cache\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.396346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.396169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lglk\" (UniqueName: \"kubernetes.io/projected/b382ed64-0983-460b-a89f-12716ddb46a7-kube-api-access-9lglk\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.396346 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.396292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-tmp-dir\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.396550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.396402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-home\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.396550 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.396431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-dshm\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.497908 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.497869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b382ed64-0983-460b-a89f-12716ddb46a7-tls-certs\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.497918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-model-cache\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.497948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lglk\" (UniqueName: \"kubernetes.io/projected/b382ed64-0983-460b-a89f-12716ddb46a7-kube-api-access-9lglk\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.497985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-tmp-dir\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.498050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-home\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498289 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.498074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-dshm\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.498439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-model-cache\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498594 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.498524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-tmp-dir\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.498656 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.498592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-home\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.500331 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.500309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-dshm\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.500554 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.500536 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b382ed64-0983-460b-a89f-12716ddb46a7-tls-certs\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.505595 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.505569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lglk\" (UniqueName: \"kubernetes.io/projected/b382ed64-0983-460b-a89f-12716ddb46a7-kube-api-access-9lglk\") pod \"auth-test-model-c17947-kserve-84554b8cd7-l9g7x\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.627413 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.627385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:12.762169 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.762139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x"] Jul 19 15:39:12.763781 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:39:12.763748 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb382ed64_0983_460b_a89f_12716ddb46a7.slice/crio-e988109b76f1dfa36fddc1b64f454e6479435e3ccfb12fc7382b0024e2018b1f WatchSource:0}: Error finding container e988109b76f1dfa36fddc1b64f454e6479435e3ccfb12fc7382b0024e2018b1f: Status 404 returned error can't find the container with id e988109b76f1dfa36fddc1b64f454e6479435e3ccfb12fc7382b0024e2018b1f Jul 19 15:39:12.814824 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:12.814793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" event={"ID":"b382ed64-0983-460b-a89f-12716ddb46a7","Type":"ContainerStarted","Data":"e988109b76f1dfa36fddc1b64f454e6479435e3ccfb12fc7382b0024e2018b1f"} Jul 19 15:39:13.821071 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:13.821029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" event={"ID":"b382ed64-0983-460b-a89f-12716ddb46a7","Type":"ContainerStarted","Data":"73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3"} Jul 19 15:39:13.839014 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:13.838966 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" podStartSLOduration=1.838949608 podStartE2EDuration="1.838949608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:39:13.8373271 +0000 UTC m=+2475.697883089" watchObservedRunningTime="2026-07-19 15:39:13.838949608 +0000 UTC m=+2475.699505595" Jul 19 15:39:22.627832 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:22.627799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:22.628317 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:22.627859 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:22.644093 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:22.644066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:39:22.874701 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:39:22.874676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:41:18.248553 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:18.248515 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr"] Jul 19 15:41:19.216083 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:19.216050 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x"] Jul 19 15:41:30.293033 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.292999 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx"] Jul 19 15:41:30.296563 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.296541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.299502 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.299479 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-51b4ec\"/\"sub-test-model-51b4ec-kserve-self-signed-certs\"" Jul 19 15:41:30.307184 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.307159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx"] Jul 19 15:41:30.347125 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.347092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-model-cache\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.347345 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.347138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tmp-dir\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.347345 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.347182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfdt\" (UniqueName: \"kubernetes.io/projected/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-kube-api-access-9qfdt\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.347345 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.347223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-dshm\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.347345 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.347253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tls-certs\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.347345 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.347309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-home\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.448766 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.448718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-model-cache\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.448766 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.448770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tmp-dir\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449027 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.448815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfdt\" (UniqueName: \"kubernetes.io/projected/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-kube-api-access-9qfdt\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449027 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.448851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-dshm\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449027 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.448896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tls-certs\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449027 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.448922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-home\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449224 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.449191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-model-cache\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449312 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.449215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tmp-dir\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.449375 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.449359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-home\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.451109 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.451085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-dshm\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.451456 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.451435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tls-certs\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.457554 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.457533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfdt\" (UniqueName: \"kubernetes.io/projected/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-kube-api-access-9qfdt\") pod \"sub-test-model-51b4ec-kserve-848576cd95-6ncnx\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.606552 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.606456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:30.741392 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:30.741362 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx"] Jul 19 15:41:30.743384 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:41:30.743355 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c4f9b81_486a_4b3d_8f5f_f7d00197d49e.slice/crio-568d08695967b0614da216c512a0bf4b9088620af3cc49f5718519187dce46bb WatchSource:0}: Error finding container 568d08695967b0614da216c512a0bf4b9088620af3cc49f5718519187dce46bb: Status 404 returned error can't find the container with id 568d08695967b0614da216c512a0bf4b9088620af3cc49f5718519187dce46bb Jul 19 15:41:31.522572 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:31.522534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" event={"ID":"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e","Type":"ContainerStarted","Data":"9900a2276be7beed66ab41e3b00f8cca5c9618de912ef42f7093d500d86430fa"} Jul 19 15:41:31.522976 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:31.522581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" event={"ID":"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e","Type":"ContainerStarted","Data":"568d08695967b0614da216c512a0bf4b9088620af3cc49f5718519187dce46bb"} Jul 19 15:41:31.540561 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:31.540502 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" podStartSLOduration=1.540485639 podStartE2EDuration="1.540485639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:41:31.538845101 +0000 UTC m=+2613.399401089" watchObservedRunningTime="2026-07-19 15:41:31.540485639 +0000 UTC m=+2613.401041627" Jul 19 15:41:33.272499 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.272456 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" podUID="8dc53d11-55f5-411a-8727-19eb83a231f5" containerName="main" containerID="cri-o://3386c487b14894d0f7d613bf953d45906eaf1f722edbb360ed255c8453615c41" gracePeriod=45 Jul 19 15:41:33.534178 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.534111 2579 generic.go:356] "Generic (PLEG): container finished" podID="8dc53d11-55f5-411a-8727-19eb83a231f5" containerID="3386c487b14894d0f7d613bf953d45906eaf1f722edbb360ed255c8453615c41" exitCode=0 Jul 19 15:41:33.534178 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.534155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" event={"ID":"8dc53d11-55f5-411a-8727-19eb83a231f5","Type":"ContainerDied","Data":"3386c487b14894d0f7d613bf953d45906eaf1f722edbb360ed255c8453615c41"} Jul 19 15:41:33.534427 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.534186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" event={"ID":"8dc53d11-55f5-411a-8727-19eb83a231f5","Type":"ContainerDied","Data":"b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c"} Jul 19 15:41:33.534427 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.534198 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b78c8ec40464443ec124ea080a56742ec93e8190ade31d57eedb0a2860f0e22c" Jul 19 15:41:33.534623 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.534606 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:41:33.578640 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578607 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-model-cache\") pod \"8dc53d11-55f5-411a-8727-19eb83a231f5\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " Jul 19 15:41:33.578805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-dshm\") pod \"8dc53d11-55f5-411a-8727-19eb83a231f5\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " Jul 19 15:41:33.578805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578690 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc53d11-55f5-411a-8727-19eb83a231f5-tls-certs\") pod \"8dc53d11-55f5-411a-8727-19eb83a231f5\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " Jul 19 15:41:33.578805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578767 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-tmp-dir\") pod \"8dc53d11-55f5-411a-8727-19eb83a231f5\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " Jul 19 15:41:33.578805 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578805 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-home\") pod \"8dc53d11-55f5-411a-8727-19eb83a231f5\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " Jul 19 15:41:33.579014 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578824 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfgmw\" (UniqueName: \"kubernetes.io/projected/8dc53d11-55f5-411a-8727-19eb83a231f5-kube-api-access-jfgmw\") pod \"8dc53d11-55f5-411a-8727-19eb83a231f5\" (UID: \"8dc53d11-55f5-411a-8727-19eb83a231f5\") " Jul 19 15:41:33.579014 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.578880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dc53d11-55f5-411a-8727-19eb83a231f5" (UID: "8dc53d11-55f5-411a-8727-19eb83a231f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:33.579136 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.579052 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dc53d11-55f5-411a-8727-19eb83a231f5" (UID: "8dc53d11-55f5-411a-8727-19eb83a231f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:33.579136 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.579083 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-home" (OuterVolumeSpecName: "home") pod "8dc53d11-55f5-411a-8727-19eb83a231f5" (UID: "8dc53d11-55f5-411a-8727-19eb83a231f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:33.579251 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.579206 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:33.579251 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.579220 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:33.579251 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.579229 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:33.580972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.580949 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc53d11-55f5-411a-8727-19eb83a231f5-kube-api-access-jfgmw" (OuterVolumeSpecName: "kube-api-access-jfgmw") pod "8dc53d11-55f5-411a-8727-19eb83a231f5" (UID: "8dc53d11-55f5-411a-8727-19eb83a231f5"). InnerVolumeSpecName "kube-api-access-jfgmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:41:33.580972 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.580960 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-dshm" (OuterVolumeSpecName: "dshm") pod "8dc53d11-55f5-411a-8727-19eb83a231f5" (UID: "8dc53d11-55f5-411a-8727-19eb83a231f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:33.581077 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.580952 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc53d11-55f5-411a-8727-19eb83a231f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dc53d11-55f5-411a-8727-19eb83a231f5" (UID: "8dc53d11-55f5-411a-8727-19eb83a231f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:41:33.679792 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.679754 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfgmw\" (UniqueName: \"kubernetes.io/projected/8dc53d11-55f5-411a-8727-19eb83a231f5-kube-api-access-jfgmw\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:33.679792 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.679787 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc53d11-55f5-411a-8727-19eb83a231f5-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:33.679792 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:33.679798 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc53d11-55f5-411a-8727-19eb83a231f5-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.236751 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.236708 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" podUID="b382ed64-0983-460b-a89f-12716ddb46a7" containerName="main" containerID="cri-o://73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3" gracePeriod=45 Jul 19 15:41:34.492557 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.492490 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:41:34.541598 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.541565 2579 generic.go:356] "Generic (PLEG): container finished" podID="b382ed64-0983-460b-a89f-12716ddb46a7" containerID="73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3" exitCode=0 Jul 19 15:41:34.541810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.541609 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" Jul 19 15:41:34.541810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.541646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" event={"ID":"b382ed64-0983-460b-a89f-12716ddb46a7","Type":"ContainerDied","Data":"73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3"} Jul 19 15:41:34.541810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.541692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x" event={"ID":"b382ed64-0983-460b-a89f-12716ddb46a7","Type":"ContainerDied","Data":"e988109b76f1dfa36fddc1b64f454e6479435e3ccfb12fc7382b0024e2018b1f"} Jul 19 15:41:34.541810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.541716 2579 scope.go:117] "RemoveContainer" containerID="73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3" Jul 19 15:41:34.541810 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.541760 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr" Jul 19 15:41:34.552778 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.551936 2579 scope.go:117] "RemoveContainer" containerID="73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3" Jul 19 15:41:34.552778 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:41:34.552240 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3\": container with ID starting with 73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3 not found: ID does not exist" containerID="73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3" Jul 19 15:41:34.552778 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.552289 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3"} err="failed to get container status \"73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3\": rpc error: code = NotFound desc = could not find container \"73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3\": container with ID starting with 73e7756b2c28732b60d993b676e29123a6b730d16a4d9849c66189f47cb23de3 not found: ID does not exist" Jul 19 15:41:34.565000 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.564975 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr"] Jul 19 15:41:34.569839 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.569818 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/auth-test-model-51b4ec-kserve-69977b5c5f-g2rrr"] Jul 19 15:41:34.587362 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-home\") pod \"b382ed64-0983-460b-a89f-12716ddb46a7\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " Jul 19 15:41:34.587476 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-tmp-dir\") pod \"b382ed64-0983-460b-a89f-12716ddb46a7\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " Jul 19 15:41:34.587476 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587400 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-dshm\") pod \"b382ed64-0983-460b-a89f-12716ddb46a7\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " Jul 19 15:41:34.587476 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587430 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b382ed64-0983-460b-a89f-12716ddb46a7-tls-certs\") pod \"b382ed64-0983-460b-a89f-12716ddb46a7\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " Jul 19 15:41:34.587476 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587446 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-model-cache\") pod \"b382ed64-0983-460b-a89f-12716ddb46a7\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " Jul 19 15:41:34.587673 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lglk\" (UniqueName: \"kubernetes.io/projected/b382ed64-0983-460b-a89f-12716ddb46a7-kube-api-access-9lglk\") pod \"b382ed64-0983-460b-a89f-12716ddb46a7\" (UID: \"b382ed64-0983-460b-a89f-12716ddb46a7\") " Jul 19 15:41:34.587730 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587678 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-home" (OuterVolumeSpecName: "home") pod "b382ed64-0983-460b-a89f-12716ddb46a7" (UID: "b382ed64-0983-460b-a89f-12716ddb46a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:34.587730 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587691 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b382ed64-0983-460b-a89f-12716ddb46a7" (UID: "b382ed64-0983-460b-a89f-12716ddb46a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:34.587838 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587825 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.587893 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587871 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.587893 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.587827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b382ed64-0983-460b-a89f-12716ddb46a7" (UID: "b382ed64-0983-460b-a89f-12716ddb46a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:34.589782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.589750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b382ed64-0983-460b-a89f-12716ddb46a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b382ed64-0983-460b-a89f-12716ddb46a7" (UID: "b382ed64-0983-460b-a89f-12716ddb46a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:41:34.589782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.589753 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-dshm" (OuterVolumeSpecName: "dshm") pod "b382ed64-0983-460b-a89f-12716ddb46a7" (UID: "b382ed64-0983-460b-a89f-12716ddb46a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:41:34.589914 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.589850 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b382ed64-0983-460b-a89f-12716ddb46a7-kube-api-access-9lglk" (OuterVolumeSpecName: "kube-api-access-9lglk") pod "b382ed64-0983-460b-a89f-12716ddb46a7" (UID: "b382ed64-0983-460b-a89f-12716ddb46a7"). InnerVolumeSpecName "kube-api-access-9lglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:41:34.613721 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.613690 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc53d11-55f5-411a-8727-19eb83a231f5" path="/var/lib/kubelet/pods/8dc53d11-55f5-411a-8727-19eb83a231f5/volumes" Jul 19 15:41:34.688709 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.688679 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.688709 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.688707 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b382ed64-0983-460b-a89f-12716ddb46a7-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.688709 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.688719 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b382ed64-0983-460b-a89f-12716ddb46a7-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.688941 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.688728 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lglk\" (UniqueName: \"kubernetes.io/projected/b382ed64-0983-460b-a89f-12716ddb46a7-kube-api-access-9lglk\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:41:34.858576 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.858497 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x"] Jul 19 15:41:34.861344 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:34.861316 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/auth-test-model-c17947-kserve-84554b8cd7-l9g7x"] Jul 19 15:41:36.613437 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:36.613396 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b382ed64-0983-460b-a89f-12716ddb46a7" path="/var/lib/kubelet/pods/b382ed64-0983-460b-a89f-12716ddb46a7/volumes" Jul 19 15:41:40.613486 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:40.613452 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:40.613946 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:40.613501 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:40.620315 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:40.620291 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:41.588628 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:41.588593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:41:50.897320 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897284 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k"] Jul 19 15:41:50.897924 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc53d11-55f5-411a-8727-19eb83a231f5" containerName="main" Jul 19 15:41:50.897924 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897736 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc53d11-55f5-411a-8727-19eb83a231f5" containerName="main" Jul 19 15:41:50.897924 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b382ed64-0983-460b-a89f-12716ddb46a7" containerName="main" Jul 19 15:41:50.897924 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897761 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b382ed64-0983-460b-a89f-12716ddb46a7" containerName="main" Jul 19 15:41:50.898150 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897946 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b382ed64-0983-460b-a89f-12716ddb46a7" containerName="main" Jul 19 15:41:50.898150 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.897975 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc53d11-55f5-411a-8727-19eb83a231f5" containerName="main" Jul 19 15:41:50.901053 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.901030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:50.905143 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.905045 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"openshift-service-ca.crt\"" Jul 19 15:41:50.905143 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.905080 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"kube-root-ca.crt\"" Jul 19 15:41:50.905143 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.905045 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"sub-test-model-c17947-kserve-self-signed-certs\"" Jul 19 15:41:50.905143 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.905145 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"default-dockercfg-zkbc6\"" Jul 19 15:41:50.914291 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:50.914246 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k"] Jul 19 15:41:51.056058 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.056024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mmf\" (UniqueName: \"kubernetes.io/projected/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-kube-api-access-58mmf\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.056291 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.056078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-home\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.056291 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.056100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tls-certs\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.056291 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.056228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-model-cache\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.056438 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.056343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tmp-dir\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.056485 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.056450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-dshm\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157525 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-model-cache\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157525 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tmp-dir\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157753 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-dshm\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157753 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mmf\" (UniqueName: \"kubernetes.io/projected/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-kube-api-access-58mmf\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157753 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-home\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157929 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tls-certs\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.157929 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.157872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-model-cache\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.158040 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.158017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-home\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.158099 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.158032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tmp-dir\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.160036 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.160010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-dshm\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.160456 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.160432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tls-certs\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.165823 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.165800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mmf\" (UniqueName: \"kubernetes.io/projected/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-kube-api-access-58mmf\") pod \"sub-test-model-c17947-kserve-7f9f74cd77-pkb9k\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.213874 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.213833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:41:51.348154 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.348121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k"] Jul 19 15:41:51.348360 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:41:51.348324 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31cd5c38_cc36_43c8_bf76_518edcfd4bc6.slice/crio-64f53433274ca8f4ad250d506aabb8ff4895bc19a9cea283aec1984deaafa2b8 WatchSource:0}: Error finding container 64f53433274ca8f4ad250d506aabb8ff4895bc19a9cea283aec1984deaafa2b8: Status 404 returned error can't find the container with id 64f53433274ca8f4ad250d506aabb8ff4895bc19a9cea283aec1984deaafa2b8 Jul 19 15:41:51.628829 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.628795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" event={"ID":"31cd5c38-cc36-43c8-bf76-518edcfd4bc6","Type":"ContainerStarted","Data":"8f7fd3a4fc9e3af064d3e10482d25b86b3eda6a4412be1de2e90e40bae03150d"} Jul 19 15:41:51.629034 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.628848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" event={"ID":"31cd5c38-cc36-43c8-bf76-518edcfd4bc6","Type":"ContainerStarted","Data":"64f53433274ca8f4ad250d506aabb8ff4895bc19a9cea283aec1984deaafa2b8"} Jul 19 15:41:51.646414 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:41:51.646352 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" podStartSLOduration=1.646335936 podStartE2EDuration="1.646335936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:41:51.645832238 +0000 UTC m=+2633.506388227" watchObservedRunningTime="2026-07-19 15:41:51.646335936 +0000 UTC m=+2633.506891923" Jul 19 15:42:01.214335 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:01.214298 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:42:01.214335 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:01.214345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:42:01.226861 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:01.226828 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:42:01.690242 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:01.690212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:42:33.685453 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:33.685405 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx"] Jul 19 15:42:34.702474 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:34.702439 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k"] Jul 19 15:42:41.981392 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:41.981354 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc"] Jul 19 15:42:41.984458 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:41.984438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:41.987077 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:41.987048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-51b4ec\"/\"rate-test-model-51b4ec-kserve-self-signed-certs\"" Jul 19 15:42:41.995078 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:41.995056 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc"] Jul 19 15:42:42.057300 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.057236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tls-certs\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.057300 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.057306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-home\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.057495 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.057366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6rg\" (UniqueName: \"kubernetes.io/projected/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-kube-api-access-6v6rg\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.057495 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.057407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tmp-dir\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.057495 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.057452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-model-cache\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.057596 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.057510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-dshm\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.158501 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tls-certs\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.158699 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-home\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.158699 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6rg\" (UniqueName: \"kubernetes.io/projected/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-kube-api-access-6v6rg\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.158699 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tmp-dir\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.158699 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-model-cache\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.158699 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-dshm\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.159035 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.158999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tmp-dir\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.159035 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.159022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-model-cache\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.159192 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.159087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-home\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.160860 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.160838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-dshm\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.161016 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.160999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tls-certs\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.166009 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.165988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6rg\" (UniqueName: \"kubernetes.io/projected/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-kube-api-access-6v6rg\") pod \"rate-test-model-51b4ec-kserve-7fbd45785f-m49hc\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.294957 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.294866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:42.426426 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.426383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc"] Jul 19 15:42:42.429779 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:42:42.429752 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df7ddd0_8fce_4e44_93ce_e6fab0a9aec2.slice/crio-530868da674ac10152d61fc78807fdd84b641112a473be17edb97e63df6df9c1 WatchSource:0}: Error finding container 530868da674ac10152d61fc78807fdd84b641112a473be17edb97e63df6df9c1: Status 404 returned error can't find the container with id 530868da674ac10152d61fc78807fdd84b641112a473be17edb97e63df6df9c1 Jul 19 15:42:42.882825 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.882789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" event={"ID":"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2","Type":"ContainerStarted","Data":"6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2"} Jul 19 15:42:42.882967 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.882838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" event={"ID":"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2","Type":"ContainerStarted","Data":"530868da674ac10152d61fc78807fdd84b641112a473be17edb97e63df6df9c1"} Jul 19 15:42:42.900602 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:42.900549 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" podStartSLOduration=1.900532297 podStartE2EDuration="1.900532297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:42:42.898687905 +0000 UTC m=+2684.759243913" watchObservedRunningTime="2026-07-19 15:42:42.900532297 +0000 UTC m=+2684.761088305" Jul 19 15:42:48.706618 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:48.706577 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" podUID="1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" containerName="main" containerID="cri-o://9900a2276be7beed66ab41e3b00f8cca5c9618de912ef42f7093d500d86430fa" gracePeriod=45 Jul 19 15:42:48.916588 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:48.916553 2579 generic.go:356] "Generic (PLEG): container finished" podID="1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" containerID="9900a2276be7beed66ab41e3b00f8cca5c9618de912ef42f7093d500d86430fa" exitCode=0 Jul 19 15:42:48.916731 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:48.916606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" event={"ID":"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e","Type":"ContainerDied","Data":"9900a2276be7beed66ab41e3b00f8cca5c9618de912ef42f7093d500d86430fa"} Jul 19 15:42:48.962880 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:48.962821 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:42:49.029597 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029566 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-home\") pod \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " Jul 19 15:42:49.029798 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-model-cache\") pod \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " Jul 19 15:42:49.029798 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029636 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tls-certs\") pod \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " Jul 19 15:42:49.029798 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029670 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-dshm\") pod \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " Jul 19 15:42:49.029798 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tmp-dir\") pod \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " Jul 19 15:42:49.029798 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029756 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qfdt\" (UniqueName: \"kubernetes.io/projected/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-kube-api-access-9qfdt\") pod \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\" (UID: \"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e\") " Jul 19 15:42:49.030062 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029867 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-home" (OuterVolumeSpecName: "home") pod "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" (UID: "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:49.030062 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.029884 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" (UID: "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:49.030175 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.030084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" (UID: "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:49.030175 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.030086 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:49.030175 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.030121 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:49.031833 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.031804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" (UID: "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:42:49.031833 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.031828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-dshm" (OuterVolumeSpecName: "dshm") pod "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" (UID: "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:49.031979 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.031858 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-kube-api-access-9qfdt" (OuterVolumeSpecName: "kube-api-access-9qfdt") pod "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" (UID: "1c4f9b81-486a-4b3d-8f5f-f7d00197d49e"). InnerVolumeSpecName "kube-api-access-9qfdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:42:49.131401 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.131364 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:49.131401 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.131398 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:49.131401 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.131407 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:49.131649 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.131416 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfdt\" (UniqueName: \"kubernetes.io/projected/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e-kube-api-access-9qfdt\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:49.724283 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.724212 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" podUID="31cd5c38-cc36-43c8-bf76-518edcfd4bc6" containerName="main" containerID="cri-o://8f7fd3a4fc9e3af064d3e10482d25b86b3eda6a4412be1de2e90e40bae03150d" gracePeriod=45 Jul 19 15:42:49.924519 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.924485 2579 generic.go:356] "Generic (PLEG): container finished" podID="31cd5c38-cc36-43c8-bf76-518edcfd4bc6" containerID="8f7fd3a4fc9e3af064d3e10482d25b86b3eda6a4412be1de2e90e40bae03150d" exitCode=0 Jul 19 15:42:49.924692 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.924565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" event={"ID":"31cd5c38-cc36-43c8-bf76-518edcfd4bc6","Type":"ContainerDied","Data":"8f7fd3a4fc9e3af064d3e10482d25b86b3eda6a4412be1de2e90e40bae03150d"} Jul 19 15:42:49.926038 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.926012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" event={"ID":"1c4f9b81-486a-4b3d-8f5f-f7d00197d49e","Type":"ContainerDied","Data":"568d08695967b0614da216c512a0bf4b9088620af3cc49f5718519187dce46bb"} Jul 19 15:42:49.926147 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.926043 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx" Jul 19 15:42:49.926147 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.926061 2579 scope.go:117] "RemoveContainer" containerID="9900a2276be7beed66ab41e3b00f8cca5c9618de912ef42f7093d500d86430fa" Jul 19 15:42:49.949429 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.949398 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx"] Jul 19 15:42:49.952442 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.952416 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/sub-test-model-51b4ec-kserve-848576cd95-6ncnx"] Jul 19 15:42:49.982883 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:49.982818 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:42:50.040201 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mmf\" (UniqueName: \"kubernetes.io/projected/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-kube-api-access-58mmf\") pod \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " Jul 19 15:42:50.040407 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040231 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-dshm\") pod \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " Jul 19 15:42:50.040407 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040302 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-home\") pod \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " Jul 19 15:42:50.040407 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tmp-dir\") pod \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " Jul 19 15:42:50.040574 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tls-certs\") pod \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " Jul 19 15:42:50.040574 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040438 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-model-cache\") pod \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\" (UID: \"31cd5c38-cc36-43c8-bf76-518edcfd4bc6\") " Jul 19 15:42:50.040667 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040646 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31cd5c38-cc36-43c8-bf76-518edcfd4bc6" (UID: "31cd5c38-cc36-43c8-bf76-518edcfd4bc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:50.040809 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040708 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-home" (OuterVolumeSpecName: "home") pod "31cd5c38-cc36-43c8-bf76-518edcfd4bc6" (UID: "31cd5c38-cc36-43c8-bf76-518edcfd4bc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:50.040897 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040799 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-model-cache" (OuterVolumeSpecName: "model-cache") pod "31cd5c38-cc36-43c8-bf76-518edcfd4bc6" (UID: "31cd5c38-cc36-43c8-bf76-518edcfd4bc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:50.040960 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040897 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:50.040960 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040917 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:50.040960 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.040930 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:50.042498 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.042467 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-dshm" (OuterVolumeSpecName: "dshm") pod "31cd5c38-cc36-43c8-bf76-518edcfd4bc6" (UID: "31cd5c38-cc36-43c8-bf76-518edcfd4bc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:42:50.042605 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.042493 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-kube-api-access-58mmf" (OuterVolumeSpecName: "kube-api-access-58mmf") pod "31cd5c38-cc36-43c8-bf76-518edcfd4bc6" (UID: "31cd5c38-cc36-43c8-bf76-518edcfd4bc6"). InnerVolumeSpecName "kube-api-access-58mmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:42:50.042605 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.042522 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31cd5c38-cc36-43c8-bf76-518edcfd4bc6" (UID: "31cd5c38-cc36-43c8-bf76-518edcfd4bc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:42:50.142090 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.142046 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:50.142090 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.142082 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mmf\" (UniqueName: \"kubernetes.io/projected/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-kube-api-access-58mmf\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:50.142090 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.142095 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cd5c38-cc36-43c8-bf76-518edcfd4bc6-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:42:50.613859 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.613827 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" path="/var/lib/kubelet/pods/1c4f9b81-486a-4b3d-8f5f-f7d00197d49e/volumes" Jul 19 15:42:50.932352 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.932322 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" Jul 19 15:42:50.932845 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.932369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k" event={"ID":"31cd5c38-cc36-43c8-bf76-518edcfd4bc6","Type":"ContainerDied","Data":"64f53433274ca8f4ad250d506aabb8ff4895bc19a9cea283aec1984deaafa2b8"} Jul 19 15:42:50.932845 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.932427 2579 scope.go:117] "RemoveContainer" containerID="8f7fd3a4fc9e3af064d3e10482d25b86b3eda6a4412be1de2e90e40bae03150d" Jul 19 15:42:50.951919 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.951875 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k"] Jul 19 15:42:50.953802 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:50.953780 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/sub-test-model-c17947-kserve-7f9f74cd77-pkb9k"] Jul 19 15:42:52.295988 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:52.295947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:52.296512 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:52.296009 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:52.308831 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:52.308800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:52.613785 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:52.613695 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31cd5c38-cc36-43c8-bf76-518edcfd4bc6" path="/var/lib/kubelet/pods/31cd5c38-cc36-43c8-bf76-518edcfd4bc6/volumes" Jul 19 15:42:52.958117 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:52.958085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:42:58.768185 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:58.768154 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:42:58.775334 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:42:58.775310 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vzhcm_0b7adfdb-fcc8-4038-8660-cfe064a9c5a7/ovn-acl-logging/0.log" Jul 19 15:43:18.400292 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.400241 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk"] Jul 19 15:43:18.400895 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.400874 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31cd5c38-cc36-43c8-bf76-518edcfd4bc6" containerName="main" Jul 19 15:43:18.400976 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.400898 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31cd5c38-cc36-43c8-bf76-518edcfd4bc6" containerName="main" Jul 19 15:43:18.400976 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.400929 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" containerName="main" Jul 19 15:43:18.400976 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.400937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" containerName="main" Jul 19 15:43:18.401096 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.401030 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4f9b81-486a-4b3d-8f5f-f7d00197d49e" containerName="main" Jul 19 15:43:18.401096 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.401045 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31cd5c38-cc36-43c8-bf76-518edcfd4bc6" containerName="main" Jul 19 15:43:18.404390 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.404354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.407189 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.407163 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"openshift-service-ca.crt\"" Jul 19 15:43:18.407328 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.407246 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"kube-root-ca.crt\"" Jul 19 15:43:18.408563 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.408543 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"default-dockercfg-zkbc6\"" Jul 19 15:43:18.408665 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.408591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c17947\"/\"rate-test-model-c17947-kserve-self-signed-certs\"" Jul 19 15:43:18.413775 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.413754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk"] Jul 19 15:43:18.501270 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.501232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-tmp-dir\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.501436 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.501331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-model-cache\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.501436 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.501359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-home\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.501436 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.501416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2kn\" (UniqueName: \"kubernetes.io/projected/f4000789-ae89-41a6-b548-db03414cfdd2-kube-api-access-pf2kn\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.501547 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.501471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-dshm\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.501547 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.501494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4000789-ae89-41a6-b548-db03414cfdd2-tls-certs\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.602653 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.602617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-tmp-dir\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.602821 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.602661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-model-cache\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.602821 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.602712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-home\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.602821 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.602783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2kn\" (UniqueName: \"kubernetes.io/projected/f4000789-ae89-41a6-b548-db03414cfdd2-kube-api-access-pf2kn\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.602934 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.602863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-dshm\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.602934 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.602902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4000789-ae89-41a6-b548-db03414cfdd2-tls-certs\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.603108 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.603087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-tmp-dir\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.603151 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.603121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-model-cache\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.603186 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.603159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-home\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.605198 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.605172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-dshm\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.605377 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.605361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4000789-ae89-41a6-b548-db03414cfdd2-tls-certs\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.611375 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.611351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2kn\" (UniqueName: \"kubernetes.io/projected/f4000789-ae89-41a6-b548-db03414cfdd2-kube-api-access-pf2kn\") pod \"rate-test-model-c17947-kserve-56f4f898b7-j8hnk\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.715179 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.715097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:18.850839 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:18.850800 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk"] Jul 19 15:43:18.853613 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:43:18.853583 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4000789_ae89_41a6_b548_db03414cfdd2.slice/crio-0df8dd61221a3910452cbd890c36b8203b86e43c3590b6e07c857c3978b40f74 WatchSource:0}: Error finding container 0df8dd61221a3910452cbd890c36b8203b86e43c3590b6e07c857c3978b40f74: Status 404 returned error can't find the container with id 0df8dd61221a3910452cbd890c36b8203b86e43c3590b6e07c857c3978b40f74 Jul 19 15:43:19.076009 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:19.075915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" event={"ID":"f4000789-ae89-41a6-b548-db03414cfdd2","Type":"ContainerStarted","Data":"2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51"} Jul 19 15:43:19.076009 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:19.075975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" event={"ID":"f4000789-ae89-41a6-b548-db03414cfdd2","Type":"ContainerStarted","Data":"0df8dd61221a3910452cbd890c36b8203b86e43c3590b6e07c857c3978b40f74"} Jul 19 15:43:19.093572 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:19.093515 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" podStartSLOduration=1.093498746 podStartE2EDuration="1.093498746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:43:19.092170771 +0000 UTC m=+2720.952726784" watchObservedRunningTime="2026-07-19 15:43:19.093498746 +0000 UTC m=+2720.954054734" Jul 19 15:43:28.716055 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:28.716013 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:28.716055 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:28.716066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:28.728617 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:28.728587 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:43:29.136624 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:43:29.136586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:44:00.774165 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:00.774124 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc"] Jul 19 15:44:01.685294 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:01.685239 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk"] Jul 19 15:44:15.794349 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:15.794282 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" podUID="6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" containerName="main" containerID="cri-o://6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2" gracePeriod=45 Jul 19 15:44:16.054408 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.054337 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:44:16.160822 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.160786 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-dshm\") pod \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " Jul 19 15:44:16.160994 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.160835 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v6rg\" (UniqueName: \"kubernetes.io/projected/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-kube-api-access-6v6rg\") pod \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " Jul 19 15:44:16.160994 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.160859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tmp-dir\") pod \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " Jul 19 15:44:16.160994 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.160884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-model-cache\") pod \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " Jul 19 15:44:16.160994 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.160914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tls-certs\") pod \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " Jul 19 15:44:16.161221 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.160996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-home\") pod \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\" (UID: \"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2\") " Jul 19 15:44:16.161221 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.161173 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" (UID: "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:16.161221 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.161182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" (UID: "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:16.161410 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.161385 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:16.161461 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.161418 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:16.161461 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.161385 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-home" (OuterVolumeSpecName: "home") pod "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" (UID: "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:16.163073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.163051 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-dshm" (OuterVolumeSpecName: "dshm") pod "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" (UID: "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:16.163073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.163060 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-kube-api-access-6v6rg" (OuterVolumeSpecName: "kube-api-access-6v6rg") pod "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" (UID: "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2"). InnerVolumeSpecName "kube-api-access-6v6rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:16.163216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.163061 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" (UID: "6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:44:16.262812 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.262763 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:16.262812 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.262800 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:16.262812 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.262813 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6rg\" (UniqueName: \"kubernetes.io/projected/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-kube-api-access-6v6rg\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:16.263115 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.262826 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:16.368524 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.368428 2579 generic.go:356] "Generic (PLEG): container finished" podID="6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" containerID="6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2" exitCode=0 Jul 19 15:44:16.368524 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.368467 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" Jul 19 15:44:16.368524 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.368492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" event={"ID":"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2","Type":"ContainerDied","Data":"6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2"} Jul 19 15:44:16.368817 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.368528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc" event={"ID":"6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2","Type":"ContainerDied","Data":"530868da674ac10152d61fc78807fdd84b641112a473be17edb97e63df6df9c1"} Jul 19 15:44:16.368817 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.368547 2579 scope.go:117] "RemoveContainer" containerID="6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2" Jul 19 15:44:16.377771 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.377750 2579 scope.go:117] "RemoveContainer" containerID="6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2" Jul 19 15:44:16.378019 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:44:16.377998 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2\": container with ID starting with 6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2 not found: ID does not exist" containerID="6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2" Jul 19 15:44:16.378067 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.378028 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2"} err="failed to get container status \"6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2\": rpc error: code = NotFound desc = could not find container \"6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2\": container with ID starting with 6d813368c1eecbbf2730340672c8887199a5794c7c28a70aefa899a4a147a8d2 not found: ID does not exist" Jul 19 15:44:16.389613 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.389592 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc"] Jul 19 15:44:16.391631 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.391608 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-51b4ec/rate-test-model-51b4ec-kserve-7fbd45785f-m49hc"] Jul 19 15:44:16.613664 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.613628 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" path="/var/lib/kubelet/pods/6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2/volumes" Jul 19 15:44:16.705603 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.705558 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" podUID="f4000789-ae89-41a6-b548-db03414cfdd2" containerName="main" containerID="cri-o://2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51" gracePeriod=45 Jul 19 15:44:16.971570 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:16.971508 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:44:17.069837 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.069798 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-dshm\") pod \"f4000789-ae89-41a6-b548-db03414cfdd2\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " Jul 19 15:44:17.069999 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.069863 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf2kn\" (UniqueName: \"kubernetes.io/projected/f4000789-ae89-41a6-b548-db03414cfdd2-kube-api-access-pf2kn\") pod \"f4000789-ae89-41a6-b548-db03414cfdd2\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " Jul 19 15:44:17.069999 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.069897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-tmp-dir\") pod \"f4000789-ae89-41a6-b548-db03414cfdd2\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " Jul 19 15:44:17.070139 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070024 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-model-cache\") pod \"f4000789-ae89-41a6-b548-db03414cfdd2\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " Jul 19 15:44:17.070139 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070054 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4000789-ae89-41a6-b548-db03414cfdd2-tls-certs\") pod \"f4000789-ae89-41a6-b548-db03414cfdd2\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " Jul 19 15:44:17.070139 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-home\") pod \"f4000789-ae89-41a6-b548-db03414cfdd2\" (UID: \"f4000789-ae89-41a6-b548-db03414cfdd2\") " Jul 19 15:44:17.070351 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070320 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4000789-ae89-41a6-b548-db03414cfdd2" (UID: "f4000789-ae89-41a6-b548-db03414cfdd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:17.070538 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070504 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4000789-ae89-41a6-b548-db03414cfdd2" (UID: "f4000789-ae89-41a6-b548-db03414cfdd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:17.070538 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070524 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-tmp-dir\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:17.070707 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.070540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-home" (OuterVolumeSpecName: "home") pod "f4000789-ae89-41a6-b548-db03414cfdd2" (UID: "f4000789-ae89-41a6-b548-db03414cfdd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:17.072111 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.072080 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4000789-ae89-41a6-b548-db03414cfdd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4000789-ae89-41a6-b548-db03414cfdd2" (UID: "f4000789-ae89-41a6-b548-db03414cfdd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:44:17.072111 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.072080 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-dshm" (OuterVolumeSpecName: "dshm") pod "f4000789-ae89-41a6-b548-db03414cfdd2" (UID: "f4000789-ae89-41a6-b548-db03414cfdd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:17.072237 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.072196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4000789-ae89-41a6-b548-db03414cfdd2-kube-api-access-pf2kn" (OuterVolumeSpecName: "kube-api-access-pf2kn") pod "f4000789-ae89-41a6-b548-db03414cfdd2" (UID: "f4000789-ae89-41a6-b548-db03414cfdd2"). InnerVolumeSpecName "kube-api-access-pf2kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:17.171887 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.171852 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-home\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:17.171887 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.171880 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-dshm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:17.171887 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.171889 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2kn\" (UniqueName: \"kubernetes.io/projected/f4000789-ae89-41a6-b548-db03414cfdd2-kube-api-access-pf2kn\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:17.172109 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.171898 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4000789-ae89-41a6-b548-db03414cfdd2-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:17.172109 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.171909 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4000789-ae89-41a6-b548-db03414cfdd2-tls-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\"" Jul 19 15:44:17.375037 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.375011 2579 generic.go:356] "Generic (PLEG): container finished" podID="f4000789-ae89-41a6-b548-db03414cfdd2" containerID="2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51" exitCode=0 Jul 19 15:44:17.375216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.375071 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" Jul 19 15:44:17.375216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.375094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" event={"ID":"f4000789-ae89-41a6-b548-db03414cfdd2","Type":"ContainerDied","Data":"2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51"} Jul 19 15:44:17.375216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.375135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk" event={"ID":"f4000789-ae89-41a6-b548-db03414cfdd2","Type":"ContainerDied","Data":"0df8dd61221a3910452cbd890c36b8203b86e43c3590b6e07c857c3978b40f74"} Jul 19 15:44:17.375216 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.375157 2579 scope.go:117] "RemoveContainer" containerID="2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51" Jul 19 15:44:17.383761 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.383713 2579 scope.go:117] "RemoveContainer" containerID="2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51" Jul 19 15:44:17.384075 ip-10-0-142-31 kubenswrapper[2579]: E0719 15:44:17.384048 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51\": container with ID starting with 2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51 not found: ID does not exist" containerID="2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51" Jul 19 15:44:17.384166 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.384080 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51"} err="failed to get container status \"2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51\": rpc error: code = NotFound desc = could not find container \"2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51\": container with ID starting with 2a062a255093465df78618bd1983820e9034c7fcafb5c966875ac1d3f5bbda51 not found: ID does not exist" Jul 19 15:44:17.396863 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.396833 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk"] Jul 19 15:44:17.403160 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:17.403137 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c17947/rate-test-model-c17947-kserve-56f4f898b7-j8hnk"] Jul 19 15:44:18.613683 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:18.613650 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4000789-ae89-41a6-b548-db03414cfdd2" path="/var/lib/kubelet/pods/f4000789-ae89-41a6-b548-db03414cfdd2/volumes" Jul 19 15:44:58.898539 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:44:58.898503 2579 scope.go:117] "RemoveContainer" containerID="3386c487b14894d0f7d613bf953d45906eaf1f722edbb360ed255c8453615c41" Jul 19 15:45:05.335635 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:05.335592 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8866d997d-pqkt2_20e71070-3d5c-420e-8ecb-f400155ff385/authorino/0.log" Jul 19 15:45:11.283230 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:11.283200 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-6d22v_02e511cb-2cec-4ee1-9965-89b08ca34aaf/copy-manifests/0.log" Jul 19 15:45:11.295768 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:11.295743 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-6d22v_02e511cb-2cec-4ee1-9965-89b08ca34aaf/manager/0.log" Jul 19 15:45:13.637362 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:13.637330 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8866d997d-pqkt2_20e71070-3d5c-420e-8ecb-f400155ff385/authorino/0.log" Jul 19 15:45:14.228556 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:14.228529 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jkl62_cb408c64-5593-4628-8576-60e02f5770c4/manager/0.log" Jul 19 15:45:15.166478 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:15.166443 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-mzvnn_1e64a97a-4be8-4958-894a-28097b5d23d2/kube-auth-proxy/0.log" Jul 19 15:45:15.403485 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:15.403447 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-skwkp_b984f288-7f6d-469c-9fb1-1eadbf63cc2c/payload-pre-processing/0.log" Jul 19 15:45:16.082094 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:16.082063 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fkn76_a68a3c4a-b0b4-4a8f-8e7f-86aa03df5021/main/0.log" Jul 19 15:45:16.196072 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:16.196041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nlrp9_6e0ffa46-206e-454c-a8ed-b2f6f4c375b2/main/0.log" Jul 19 15:45:16.537021 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:16.536991 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pz97v_6f7c7987-4a0e-4b4c-8d74-26280b5ced00/main/0.log" Jul 19 15:45:21.905393 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905357 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h7n64/must-gather-mcn4l"] Jul 19 15:45:21.905781 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905740 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4000789-ae89-41a6-b548-db03414cfdd2" containerName="main" Jul 19 15:45:21.905781 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905755 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4000789-ae89-41a6-b548-db03414cfdd2" containerName="main" Jul 19 15:45:21.905781 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905777 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" containerName="main" Jul 19 15:45:21.905943 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905786 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" containerName="main" Jul 19 15:45:21.905943 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905876 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4000789-ae89-41a6-b548-db03414cfdd2" containerName="main" Jul 19 15:45:21.905943 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.905906 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df7ddd0-8fce-4e44-93ce-e6fab0a9aec2" containerName="main" Jul 19 15:45:21.909457 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.909435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:21.912379 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.912358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h7n64\"/\"kube-root-ca.crt\"" Jul 19 15:45:21.912513 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.912356 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h7n64\"/\"openshift-service-ca.crt\"" Jul 19 15:45:21.913616 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.913593 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h7n64\"/\"default-dockercfg-7hpkc\"" Jul 19 15:45:21.919054 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.919031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/must-gather-mcn4l"] Jul 19 15:45:21.986834 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.986797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e91000c-6ad8-480f-bc39-7701cef7f891-must-gather-output\") pod \"must-gather-mcn4l\" (UID: \"9e91000c-6ad8-480f-bc39-7701cef7f891\") " pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:21.986834 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:21.986836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26g4d\" (UniqueName: \"kubernetes.io/projected/9e91000c-6ad8-480f-bc39-7701cef7f891-kube-api-access-26g4d\") pod \"must-gather-mcn4l\" (UID: \"9e91000c-6ad8-480f-bc39-7701cef7f891\") " pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:22.087636 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.087602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e91000c-6ad8-480f-bc39-7701cef7f891-must-gather-output\") pod \"must-gather-mcn4l\" (UID: \"9e91000c-6ad8-480f-bc39-7701cef7f891\") " pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:22.087636 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.087640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26g4d\" (UniqueName: \"kubernetes.io/projected/9e91000c-6ad8-480f-bc39-7701cef7f891-kube-api-access-26g4d\") pod \"must-gather-mcn4l\" (UID: \"9e91000c-6ad8-480f-bc39-7701cef7f891\") " pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:22.088016 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.087995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e91000c-6ad8-480f-bc39-7701cef7f891-must-gather-output\") pod \"must-gather-mcn4l\" (UID: \"9e91000c-6ad8-480f-bc39-7701cef7f891\") " pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:22.095268 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.095237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26g4d\" (UniqueName: \"kubernetes.io/projected/9e91000c-6ad8-480f-bc39-7701cef7f891-kube-api-access-26g4d\") pod \"must-gather-mcn4l\" (UID: \"9e91000c-6ad8-480f-bc39-7701cef7f891\") " pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:22.219785 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.219688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/must-gather-mcn4l" Jul 19 15:45:22.353955 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.353932 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/must-gather-mcn4l"] Jul 19 15:45:22.356905 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:45:22.356876 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e91000c_6ad8_480f_bc39_7701cef7f891.slice/crio-e28a9932f65cff25e91481325835299bc8010c3c36504f2a87a6181f22f3c456 WatchSource:0}: Error finding container e28a9932f65cff25e91481325835299bc8010c3c36504f2a87a6181f22f3c456: Status 404 returned error can't find the container with id e28a9932f65cff25e91481325835299bc8010c3c36504f2a87a6181f22f3c456 Jul 19 15:45:22.358881 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.358861 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:45:22.686223 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:22.686189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/must-gather-mcn4l" event={"ID":"9e91000c-6ad8-480f-bc39-7701cef7f891","Type":"ContainerStarted","Data":"e28a9932f65cff25e91481325835299bc8010c3c36504f2a87a6181f22f3c456"} Jul 19 15:45:23.707209 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:23.707161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/must-gather-mcn4l" event={"ID":"9e91000c-6ad8-480f-bc39-7701cef7f891","Type":"ContainerStarted","Data":"2ee95ade6347ceb97573da8a5e2f1796ac0b79fbfc1898727ce5b68bccb0fc14"} Jul 19 15:45:23.707700 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:23.707225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/must-gather-mcn4l" event={"ID":"9e91000c-6ad8-480f-bc39-7701cef7f891","Type":"ContainerStarted","Data":"2ea4e289bb03792a0a5a78ac43f1206b60438fe7832f4f68648ff6427f1a5259"} Jul 19 15:45:23.723588 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:23.723522 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h7n64/must-gather-mcn4l" podStartSLOduration=1.8841019430000001 podStartE2EDuration="2.723500471s" totalImagesPullingTime="839.398528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:45:23.721551053 +0000 UTC m=+2845.582107041" watchObservedRunningTime="2026-07-19 15:45:23.723500471 +0000 UTC m=+2845.584056461" Jul 19 15:45:24.704754 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:24.704724 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mg8z6_9cadb2b0-9a6e-4d64-8d1e-8b5bb0aefe21/global-pull-secret-syncer/0.log" Jul 19 15:45:24.844152 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:24.844123 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vshh6_6f8d969b-9bc6-4405-8067-f503b71bf9d0/konnectivity-agent/0.log" Jul 19 15:45:24.909855 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:24.909815 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-31.ec2.internal_a290a0755b8688c1e5c83007894063ed/haproxy/0.log" Jul 19 15:45:30.085288 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:30.084495 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8866d997d-pqkt2_20e71070-3d5c-420e-8ecb-f400155ff385/authorino/0.log" Jul 19 15:45:30.347591 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:30.346746 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jkl62_cb408c64-5593-4628-8576-60e02f5770c4/manager/0.log" Jul 19 15:45:31.882121 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:31.882038 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/alertmanager/0.log" Jul 19 15:45:31.910986 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:31.910937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/config-reloader/0.log" Jul 19 15:45:31.933278 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:31.933235 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/kube-rbac-proxy-web/0.log" Jul 19 15:45:31.952612 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:31.952585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/kube-rbac-proxy/0.log" Jul 19 15:45:31.972350 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:31.972322 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/kube-rbac-proxy-metric/0.log" Jul 19 15:45:31.993384 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:31.993294 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/prom-label-proxy/0.log" Jul 19 15:45:32.013539 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.013513 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6e487a7c-f0e2-4661-bff6-c350fdac8bb7/init-config-reloader/0.log" Jul 19 15:45:32.095310 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.095279 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-n9kmq_fc6a2f39-a519-474e-889c-28bbfd94bf4e/kube-state-metrics/0.log" Jul 19 15:45:32.132079 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.132052 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-n9kmq_fc6a2f39-a519-474e-889c-28bbfd94bf4e/kube-rbac-proxy-main/0.log" Jul 19 15:45:32.154149 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.154072 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-n9kmq_fc6a2f39-a519-474e-889c-28bbfd94bf4e/kube-rbac-proxy-self/0.log" Jul 19 15:45:32.183499 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.183473 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5cdfb59d7f-kpnwt_7eb8b6b0-eb43-4396-a251-3f7cbe554336/metrics-server/0.log" Jul 19 15:45:32.210916 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.210878 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-9tppl_fe0a3e92-266d-4de2-86db-4efdf0c82312/monitoring-plugin/0.log" Jul 19 15:45:32.359709 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.359676 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cd8nd_b5b7f146-7844-4978-b6ea-e2e61559fe5a/node-exporter/0.log" Jul 19 15:45:32.386766 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.386728 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cd8nd_b5b7f146-7844-4978-b6ea-e2e61559fe5a/kube-rbac-proxy/0.log" Jul 19 15:45:32.410281 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.408175 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cd8nd_b5b7f146-7844-4978-b6ea-e2e61559fe5a/init-textfile/0.log" Jul 19 15:45:32.523792 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.523765 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-sl8v4_a7367334-c047-4877-a867-b9fd05c218ac/kube-rbac-proxy-main/0.log" Jul 19 15:45:32.551289 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.551194 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-sl8v4_a7367334-c047-4877-a867-b9fd05c218ac/kube-rbac-proxy-self/0.log" Jul 19 15:45:32.572603 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.572573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-sl8v4_a7367334-c047-4877-a867-b9fd05c218ac/openshift-state-metrics/0.log" Jul 19 15:45:32.619919 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.619878 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/prometheus/0.log" Jul 19 15:45:32.645183 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.645129 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/config-reloader/0.log" Jul 19 15:45:32.669613 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.669582 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/thanos-sidecar/0.log" Jul 19 15:45:32.689397 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.689364 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/kube-rbac-proxy-web/0.log" Jul 19 15:45:32.709339 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.709308 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/kube-rbac-proxy/0.log" Jul 19 15:45:32.731712 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.731683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/kube-rbac-proxy-thanos/0.log" Jul 19 15:45:32.753377 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.753346 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7de23c68-0731-4681-8251-88ad538ab4d2/init-config-reloader/0.log" Jul 19 15:45:32.784855 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.784820 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-br69q_caf614ce-4614-4ac4-9884-3c0a32f512c0/prometheus-operator/0.log" Jul 19 15:45:32.801236 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.801203 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-br69q_caf614ce-4614-4ac4-9884-3c0a32f512c0/kube-rbac-proxy/0.log" Jul 19 15:45:32.827366 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.827278 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-ttjqq_4de4a976-1d92-4780-8c1b-f6e976d307ee/prometheus-operator-admission-webhook/0.log" Jul 19 15:45:32.949313 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.949281 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db44bb49-2slcq_0c17fba3-b20b-4128-b203-d96fa6cb5aea/thanos-query/0.log" Jul 19 15:45:32.972429 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.972396 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db44bb49-2slcq_0c17fba3-b20b-4128-b203-d96fa6cb5aea/kube-rbac-proxy-web/0.log" Jul 19 15:45:32.991990 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:32.991961 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db44bb49-2slcq_0c17fba3-b20b-4128-b203-d96fa6cb5aea/kube-rbac-proxy/0.log" Jul 19 15:45:33.012289 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:33.012241 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db44bb49-2slcq_0c17fba3-b20b-4128-b203-d96fa6cb5aea/prom-label-proxy/0.log" Jul 19 15:45:33.031969 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:33.031909 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db44bb49-2slcq_0c17fba3-b20b-4128-b203-d96fa6cb5aea/kube-rbac-proxy-rules/0.log" Jul 19 15:45:33.052470 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:33.052433 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78db44bb49-2slcq_0c17fba3-b20b-4128-b203-d96fa6cb5aea/kube-rbac-proxy-metrics/0.log" Jul 19 15:45:35.089073 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.089032 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8"] Jul 19 15:45:35.094241 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.094211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.108900 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.108869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8"] Jul 19 15:45:35.255556 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.255520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-podres\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.255754 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.255624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297td\" (UniqueName: \"kubernetes.io/projected/4800810a-d0fc-496a-a6a2-9f9566b1278c-kube-api-access-297td\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.255754 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.255643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-proc\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.255754 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.255665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-lib-modules\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.255754 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.255713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-sys\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.356574 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-podres\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.356746 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297td\" (UniqueName: \"kubernetes.io/projected/4800810a-d0fc-496a-a6a2-9f9566b1278c-kube-api-access-297td\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.356746 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-proc\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.356746 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-lib-modules\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.356746 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-podres\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.356746 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-sys\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.357029 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-sys\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.357029 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-lib-modules\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.357155 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.356775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4800810a-d0fc-496a-a6a2-9f9566b1278c-proc\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.366646 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.366605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-297td\" (UniqueName: \"kubernetes.io/projected/4800810a-d0fc-496a-a6a2-9f9566b1278c-kube-api-access-297td\") pod \"perf-node-gather-daemonset-lmtb8\" (UID: \"4800810a-d0fc-496a-a6a2-9f9566b1278c\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.416660 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.416625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.592276 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.588907 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5578bc4bdc-g8bkg_169c8c15-208c-4617-b1e5-95cb56ac0f2e/console/0.log" Jul 19 15:45:35.603161 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.603128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8"] Jul 19 15:45:35.606950 ip-10-0-142-31 kubenswrapper[2579]: W0719 15:45:35.606874 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4800810a_d0fc_496a_a6a2_9f9566b1278c.slice/crio-1e912704f94980ee7706782f46703a9d56da0abcc738c809b00902247ca37841 WatchSource:0}: Error finding container 1e912704f94980ee7706782f46703a9d56da0abcc738c809b00902247ca37841: Status 404 returned error can't find the container with id 1e912704f94980ee7706782f46703a9d56da0abcc738c809b00902247ca37841 Jul 19 15:45:35.794976 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.794926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" event={"ID":"4800810a-d0fc-496a-a6a2-9f9566b1278c","Type":"ContainerStarted","Data":"e9ea2819e00e0b853728028e9df72a616d32cecf2cfdeae20b7102d91f65eb7a"} Jul 19 15:45:35.795170 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.795007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" event={"ID":"4800810a-d0fc-496a-a6a2-9f9566b1278c","Type":"ContainerStarted","Data":"1e912704f94980ee7706782f46703a9d56da0abcc738c809b00902247ca37841"} Jul 19 15:45:35.795881 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.795849 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:35.812815 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:35.812743 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" podStartSLOduration=0.812718948 podStartE2EDuration="812.718948ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 15:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:45:35.812317143 +0000 UTC m=+2857.672873133" watchObservedRunningTime="2026-07-19 15:45:35.812718948 +0000 UTC m=+2857.673274990" Jul 19 15:45:37.269090 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:37.269064 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94kz9_30103e94-7686-45ce-8ab5-a34c3a793cf5/dns/0.log" Jul 19 15:45:37.291187 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:37.291140 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94kz9_30103e94-7686-45ce-8ab5-a34c3a793cf5/kube-rbac-proxy/0.log" Jul 19 15:45:37.414797 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:37.414769 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bc55h_2ca150b8-5079-4fb5-99b1-4133ba4ae43a/dns-node-resolver/0.log" Jul 19 15:45:38.021851 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:38.021803 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x666x_8717acab-00c9-46d6-b88b-fcc123b04b9c/node-ca/0.log" Jul 19 15:45:39.074070 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:39.074035 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-mzvnn_1e64a97a-4be8-4958-894a-28097b5d23d2/kube-auth-proxy/0.log" Jul 19 15:45:39.158981 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:39.158935 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-skwkp_b984f288-7f6d-469c-9fb1-1eadbf63cc2c/payload-pre-processing/0.log" Jul 19 15:45:39.886669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:39.886645 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gpj89_f449b873-72ee-4366-97ab-65026aba1100/serve-healthcheck-canary/0.log" Jul 19 15:45:40.381669 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:40.381640 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8jltm_e86b30e8-a3f4-425b-84e7-8fa8f7bf6531/kube-rbac-proxy/0.log" Jul 19 15:45:40.403543 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:40.403514 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8jltm_e86b30e8-a3f4-425b-84e7-8fa8f7bf6531/exporter/0.log" Jul 19 15:45:40.422369 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:40.422311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8jltm_e86b30e8-a3f4-425b-84e7-8fa8f7bf6531/extractor/0.log" Jul 19 15:45:42.821500 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:42.821466 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-lmtb8" Jul 19 15:45:43.058398 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:43.058349 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-6d22v_02e511cb-2cec-4ee1-9965-89b08ca34aaf/manager/0.log" Jul 19 15:45:43.074094 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:43.074027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-6d22v_02e511cb-2cec-4ee1-9965-89b08ca34aaf/copy-manifests/0.log" Jul 19 15:45:44.525582 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:44.525552 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57c99ccc67-9lxl5_fbb79277-c8e8-4dbb-8cd4-0574150d6cc5/manager/0.log" Jul 19 15:45:46.012521 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:46.012484 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-mzvnn_1e64a97a-4be8-4958-894a-28097b5d23d2/kube-auth-proxy/0.log" Jul 19 15:45:46.084123 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:46.084075 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-skwkp_b984f288-7f6d-469c-9fb1-1eadbf63cc2c/payload-pre-processing/0.log" Jul 19 15:45:47.497747 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:47.497684 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-mzvnn_1e64a97a-4be8-4958-894a-28097b5d23d2/kube-auth-proxy/0.log" Jul 19 15:45:47.570656 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:47.570620 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-skwkp_b984f288-7f6d-469c-9fb1-1eadbf63cc2c/payload-pre-processing/0.log" Jul 19 15:45:51.057782 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.057697 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9g5fl_172caf8f-fd5d-4508-962d-ee98378bf734/kube-multus/0.log" Jul 19 15:45:51.080418 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.080394 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/kube-multus-additional-cni-plugins/0.log" Jul 19 15:45:51.098424 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.098395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/egress-router-binary-copy/0.log" Jul 19 15:45:51.117081 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.117054 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/cni-plugins/0.log" Jul 19 15:45:51.135350 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.135318 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/bond-cni-plugin/0.log" Jul 19 15:45:51.153808 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.153788 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/routeoverride-cni/0.log" Jul 19 15:45:51.172301 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.172244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/whereabouts-cni-bincopy/0.log" Jul 19 15:45:51.189966 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.189944 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kfl7_69db3133-e92e-47d0-9a7b-38d6501694b9/whereabouts-cni/0.log" Jul 19 15:45:51.595335 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.595302 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tc8s7_b9f28b58-145b-4dcb-a3b2-82741e2ad378/network-metrics-daemon/0.log" Jul 19 15:45:51.612167 ip-10-0-142-31 kubenswrapper[2579]: I0719 15:45:51.612141 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tc8s7_b9f28b58-145b-4dcb-a3b2-82741e2ad378/kube-rbac-proxy/0.log"