Jul 09 16:13:18.781936 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 16:13:18.781947 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 16:13:18.781955 ip-10-0-140-120 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 16:13:18.782209 ip-10-0-140-120 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 16:13:28.624750 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 16:13:28.624765 ip-10-0-140-120 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8a47b0832c22428e870336e55a908eae -- Jul 09 16:15:22.061032 ip-10-0-140-120 systemd[1]: Starting Kubernetes Kubelet... Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.545812 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552207 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552220 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552224 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552227 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552231 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552234 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552237 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:22.729714 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552240 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:22.631567 ip-10-0-140-120 systemd[1]: Started Kubernetes Kubelet. Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552245 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552249 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552253 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552257 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552264 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552267 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552271 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552274 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552277 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552280 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552282 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552285 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552288 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552291 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552294 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552297 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552300 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552303 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552306 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:22.730830 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552308 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552311 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552315 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552318 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552320 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552323 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552326 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552328 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552331 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552334 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552337 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552340 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552343 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552346 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552349 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552352 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552355 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552358 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552361 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:22.731552 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552363 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552366 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552369 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552371 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552374 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552376 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552379 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552382 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552385 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552387 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552391 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552394 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552397 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552400 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552403 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552406 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552409 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552412 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552415 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552417 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:22.732225 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552420 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552423 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552425 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552428 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552433 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552438 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552441 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552444 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552447 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552449 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552452 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552455 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552457 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552460 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552462 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552465 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552468 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552471 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552474 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:22.732822 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552476 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552479 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552483 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552488 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552491 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552494 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552497 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552500 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552503 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552506 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552508 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552511 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552513 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.552516 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553313 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553322 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553326 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553329 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553332 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553336 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:22.733480 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553339 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553341 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553344 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553348 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553351 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553354 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553356 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553359 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553362 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553364 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553367 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553370 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553372 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553375 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553377 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553381 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553384 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553387 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553390 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:22.734104 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553392 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553395 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553397 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553400 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553403 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553405 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553408 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553411 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553413 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553416 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553419 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553422 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553424 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553429 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553432 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553435 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553441 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553445 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553448 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:22.734765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553452 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553456 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553460 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553463 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553466 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553469 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553472 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553475 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553478 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553481 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553483 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553487 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553490 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553493 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553495 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553498 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553501 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553503 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553506 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:22.735351 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553508 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553511 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553514 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553517 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553520 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553522 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553526 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553530 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553536 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553539 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553542 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553545 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553547 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553550 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553553 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553556 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553558 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553561 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553564 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:22.736007 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553566 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553569 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553571 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553574 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553576 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553579 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553582 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553584 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553587 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553589 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553592 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553595 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553597 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553600 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553602 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553605 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.553607 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554060 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554066 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554070 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554073 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:22.736587 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554077 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554081 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554085 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554088 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554091 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554097 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554100 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554103 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554106 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554109 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554112 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554114 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554117 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554120 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554122 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554125 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554128 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554131 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554134 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554136 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:22.737260 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554139 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554142 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554144 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554147 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554150 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554152 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554155 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554158 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554160 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554163 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554166 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554169 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554171 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554175 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554178 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554180 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554184 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554190 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554193 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:22.737852 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554196 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554201 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554204 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554207 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554211 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554214 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554217 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554220 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554222 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554225 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554228 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554231 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554234 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554236 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554239 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554241 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554244 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554247 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554250 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:22.738432 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554252 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554255 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554258 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554260 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554263 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554265 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554268 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554273 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554278 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554281 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554284 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554289 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554292 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554294 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554297 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554300 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554302 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554305 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554308 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:22.738940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554310 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554313 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554316 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554319 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554321 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554324 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554327 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554329 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554332 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554335 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554337 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554340 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554343 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554345 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554348 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554351 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554353 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.554356 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554425 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554432 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554438 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 16:15:22.739508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554444 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554450 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554453 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554459 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554463 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554467 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554470 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554473 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554477 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554480 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554484 2562 flags.go:64] FLAG: --cgroup-root="" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554487 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554490 2562 flags.go:64] FLAG: --client-ca-file="" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554493 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554496 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554500 2562 flags.go:64] FLAG: --cluster-domain="" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554504 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554507 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554510 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554514 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554522 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554525 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554528 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 16:15:22.740073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554532 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554535 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554539 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554542 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554545 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554548 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554551 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554554 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554557 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554562 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554565 2562 flags.go:64] FLAG: --enable-server="true" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554569 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554575 2562 flags.go:64] FLAG: --event-burst="100" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554578 2562 flags.go:64] FLAG: --event-qps="50" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554581 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554584 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554587 2562 flags.go:64] FLAG: --eviction-hard="" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554591 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554594 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554597 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554601 2562 flags.go:64] FLAG: --eviction-soft="" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554604 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554607 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554610 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554612 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 16:15:22.740713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554615 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554619 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554621 2562 flags.go:64] FLAG: --feature-gates="" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554625 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554628 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554631 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554635 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554638 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554642 2562 flags.go:64] FLAG: --help="false" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554645 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554648 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554651 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554654 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554658 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554661 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554664 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554668 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554671 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554674 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554679 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554683 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554686 2562 flags.go:64] FLAG: --kube-reserved="" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554689 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554692 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 16:15:23.018493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554695 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554698 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554701 2562 flags.go:64] FLAG: --lock-file="" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554704 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554707 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554710 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554716 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554719 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554721 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554724 2562 flags.go:64] FLAG: --logging-format="text" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554728 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554731 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554734 2562 flags.go:64] FLAG: --manifest-url="" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554737 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554741 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554744 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554748 2562 flags.go:64] FLAG: --max-pods="110" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554751 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554754 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554757 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554760 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554763 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554766 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554778 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554787 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 16:15:23.044742 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554790 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554793 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554798 2562 flags.go:64] FLAG: --pod-cidr="" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554804 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554810 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554813 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554817 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554819 2562 flags.go:64] FLAG: --port="10250" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554823 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554826 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e57585f290ece44" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554830 2562 flags.go:64] FLAG: --qos-reserved="" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554833 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554836 2562 flags.go:64] FLAG: --register-node="true" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554839 2562 flags.go:64] FLAG: --register-with-taints="" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554842 2562 flags.go:64] FLAG: --registry-burst="10" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554845 2562 flags.go:64] FLAG: --registry-qps="5" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554848 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554851 2562 flags.go:64] FLAG: --reserved-memory="" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554855 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554858 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554861 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554864 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554866 2562 flags.go:64] FLAG: --runonce="false" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554869 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554872 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 16:15:23.047261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554875 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554891 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554895 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554899 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554902 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554905 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554909 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554912 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554915 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554918 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554923 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554926 2562 flags.go:64] FLAG: --system-cgroups="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554930 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554936 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554938 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554941 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554946 2562 flags.go:64] FLAG: --tls-min-version="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554949 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554969 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554973 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554976 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554979 2562 flags.go:64] FLAG: --v="2" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554983 2562 flags.go:64] FLAG: --version="false" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554987 2562 flags.go:64] FLAG: --vmodule="" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554991 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 16:15:23.048125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.554994 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555117 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555121 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555124 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555128 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555130 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555133 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555137 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555139 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555142 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555145 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555148 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555151 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555155 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555158 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555160 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555163 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555167 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555170 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555173 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:23.048921 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555176 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555179 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555181 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555184 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555187 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555189 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555192 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555195 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555197 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555200 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555203 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555205 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555208 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555211 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555213 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555216 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555219 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555221 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555224 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555227 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:23.049561 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555229 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555232 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555235 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555237 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555240 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555244 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555247 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555250 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555252 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555257 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555259 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555262 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555267 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555270 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555273 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555276 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555279 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555282 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555285 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:23.050235 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555288 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555291 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555294 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555298 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555301 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555303 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555306 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555309 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555311 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555314 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555317 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555320 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555324 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555327 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555330 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555333 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555336 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555339 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555343 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:23.050766 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555346 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555349 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555351 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555355 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555358 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555361 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555363 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555366 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555369 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555372 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555375 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555378 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555382 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555385 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555388 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555390 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555393 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555396 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555398 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555401 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:23.051734 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555404 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.555407 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.555412 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.562583 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.562599 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.562620 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.562626 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562701 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562707 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562710 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562713 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562716 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562719 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562721 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562724 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:23.052314 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562727 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562730 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562733 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562736 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562738 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562742 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562744 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562747 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562750 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562753 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562756 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562758 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562761 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562764 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562767 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562769 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562773 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562776 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562778 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562781 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:23.345858 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562784 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562787 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562790 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562793 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562796 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562798 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562801 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562804 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562807 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562809 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562812 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562814 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562820 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562823 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562825 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562828 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562830 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562834 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562836 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562839 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:23.347331 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562842 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562845 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562847 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562850 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562853 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562855 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562858 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562861 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562864 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562868 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562873 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562877 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562879 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562882 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562886 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562889 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562892 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562895 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562897 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:23.348013 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562900 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562904 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562908 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562911 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562914 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562919 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562923 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562926 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562929 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562932 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562934 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562937 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562940 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562943 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562945 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562948 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562951 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562977 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562981 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562983 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:23.349152 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562986 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562989 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562992 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562995 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.562998 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563003 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563006 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563009 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563015 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563018 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563021 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563024 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.563029 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563149 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:23.349767 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563154 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563158 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563162 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563166 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563170 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563173 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563176 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563179 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563181 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563184 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563187 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563190 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563192 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563195 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563198 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563200 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563203 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563205 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563209 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563213 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:23.350236 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563216 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563219 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563222 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563224 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563227 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563230 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563233 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563238 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563240 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563243 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563246 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563248 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563251 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563253 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563257 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563262 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563265 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563268 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563270 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:23.350986 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563273 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563276 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563279 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563281 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563284 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563287 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563290 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563292 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563295 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563297 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563300 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563302 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563305 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563308 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563310 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563313 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563315 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563318 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563320 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563323 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:23.351565 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563327 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563331 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563333 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563336 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563339 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563341 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563344 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563346 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563351 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563354 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563357 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563360 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563364 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563387 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563391 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563394 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563397 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563400 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563403 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:23.352180 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563406 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563409 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563412 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563414 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563417 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563419 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563422 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563425 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563428 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563430 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563433 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563435 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563438 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563442 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563445 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563449 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563451 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563454 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563457 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:23.352972 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:22.563460 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.563465 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.564135 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.569459 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.571104 2562 server.go:1019] "Starting client certificate rotation" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.571203 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.571239 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.575309 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.578731 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.589845 2562 log.go:25] "Validated CRI v1 runtime API" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.593870 2562 log.go:25] "Validated CRI v1 image API" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.594725 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 16:15:23.353568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.599751 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 16:15:23.354074 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.600845 2562 fs.go:135] Filesystem UUIDs: map[1515af79-3869-41cb-8e53-7d3c6bf2d1bb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d45ab21d-b4a9-4156-a78f-4ce650d25ef4:/dev/nvme0n1p3] Jul 09 16:15:23.354074 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.600861 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.607201 2562 manager.go:216] Machine: {Timestamp:2026-07-09 16:15:22.605762613 +0000 UTC m=+0.403983171 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199966 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d50c26650962ab0a503592239503b SystemUUID:ec2d50c2-6650-962a-b0a5-03592239503b BootID:8a47b083-2c22-428e-8703-36e55a908eae Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:74:e5:36:a0:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:74:e5:36:a0:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:6b:dc:e1:a4:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.607303 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.607379 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.610385 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.610413 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.611171 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.611181 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.611194 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.611205 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.612870 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617739 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617898 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617927 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617937 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617972 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617984 2562 kubelet.go:398] "Adding apiserver pod source" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.617994 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 16:15:23.639481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.618339 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.618354 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.621864 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmdkw" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.622173 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624546 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624576 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624722 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624735 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624741 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624747 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624755 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.624761 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.625456 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.625463 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.625470 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.625476 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.625485 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.625494 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.626413 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.626421 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.629438 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmdkw" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.629573 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.629589 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.629818 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.630641 2562 server.go:1268] "Started kubelet" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.630848 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.630920 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.631154 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.632154 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.633651 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.637155 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.637166 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.637250 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.636352 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18c0ab3b11e8dc75 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-07-09 16:15:22.630610037 +0000 UTC m=+0.428830572,LastTimestamp:2026-07-09 16:15:22.630610037 +0000 UTC m=+0.428830572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.637835 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.637879 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.637895 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.637979 2562 factory.go:55] Registering systemd factory Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638016 2562 factory.go:223] Registration of the systemd container factory successfully Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638032 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 09 16:15:23.640295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638039 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.638232 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638293 2562 factory.go:153] Registering CRI-O factory Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638304 2562 factory.go:223] Registration of the crio container factory successfully Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638358 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638387 2562 factory.go:103] Registering Raw factory Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638405 2562 manager.go:1196] Started watching for new ooms in manager Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.638988 2562 manager.go:349] Starting recovery of all containers Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.642709 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.645107 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-120.ec2.internal\" not found" node="ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.651592 2562 manager.go:354] Recovery completed Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.658789 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.661947 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.661976 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.661989 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.662520 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.662530 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.662539 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.662547 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.663245 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.663269 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.663280 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.664905 2562 policy_none.go:49] "None policy: Start" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.664919 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.664928 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.667060 2562 policy_none.go:47] "Start" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.700725 2562 manager.go:337] "Starting Device Plugin manager" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.700758 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.700771 2562 server.go:87] "Starting device plugin registration server" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.701051 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.701066 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.701157 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.701232 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.701240 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.703074 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.703129 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.802065 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.803098 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.803131 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.803146 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.803178 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.809684 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.809704 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-120.ec2.internal\": node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.812412 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.813604 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.813642 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.813665 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.813696 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 16:15:23.641634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.815628 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.838443 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.918295 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"] Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.918383 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.919376 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.919408 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.919418 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.921971 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922042 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922069 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922770 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922788 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922797 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922829 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922846 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.922856 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.925126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.925150 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.925777 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.925804 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:22.925815 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.938687 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.958731 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.ec2.internal\" not found" node="ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:22.963319 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.ec2.internal\" not found" node="ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.039659 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.039717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aa83e0abbce66175145d3331b72251e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"0aa83e0abbce66175145d3331b72251e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.039741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa83e0abbce66175145d3331b72251e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"0aa83e0abbce66175145d3331b72251e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.039760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09061521089a77af38194f9d3efcc858-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.ec2.internal\" (UID: \"09061521089a77af38194f9d3efcc858\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.140436 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.140505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aa83e0abbce66175145d3331b72251e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"0aa83e0abbce66175145d3331b72251e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.643254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.140533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa83e0abbce66175145d3331b72251e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"0aa83e0abbce66175145d3331b72251e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.140551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09061521089a77af38194f9d3efcc858-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.ec2.internal\" (UID: \"09061521089a77af38194f9d3efcc858\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.140587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09061521089a77af38194f9d3efcc858-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.ec2.internal\" (UID: \"09061521089a77af38194f9d3efcc858\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.140595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0aa83e0abbce66175145d3331b72251e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"0aa83e0abbce66175145d3331b72251e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.140610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa83e0abbce66175145d3331b72251e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"0aa83e0abbce66175145d3331b72251e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.244635 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.260976 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.267092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.345727 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.446275 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.546389 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.546424 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.571210 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.571318 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.571349 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.571375 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.618977 2562 apiserver.go:52] "Watching apiserver" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.634355 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 16:10:23 +0000 UTC" deadline="2028-01-19 12:34:55.059907459 +0000 UTC" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.634400 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13412h19m31.425511408s" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.637314 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.637317 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.641551 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.641947 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 16:15:23.644169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.642321 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-b8dkn","kube-system/konnectivity-agent-wpqxt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq","openshift-cluster-node-tuning-operator/tuned-8wl75","openshift-dns/node-resolver-d6m5b","openshift-image-registry/node-ca-xbgpm","openshift-multus/multus-additional-cni-plugins-hrjzl","openshift-network-diagnostics/network-check-target-xlhnm","openshift-multus/multus-k8xpd","openshift-multus/network-metrics-daemon-mpbbn","openshift-network-operator/iptables-alerter-4rzl5","openshift-ovn-kubernetes/ovnkube-node-s8c2d"] Jul 09 16:15:23.645028 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.644940 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.645028 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.644998 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.647336 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.648343 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.649562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.651117 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.651258 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.651380 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.651414 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tjl8g\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.651715 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.652458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.654619 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.654646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.654830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.656825 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.656936 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m299r\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.657403 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.659189 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.659364 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f2vcp\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.659849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.659930 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660558 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660637 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660650 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tvqd9\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660640 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660945 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660950 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2d6p4\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.660964 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.662194 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.662204 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.662261 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.663283 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.663283 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.663594 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.663646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dgcfw\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.663654 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.663594 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.897994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.664570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.666926 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.666979 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.667540 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.667544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x5g2b\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.669199 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.671636 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"] Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.671664 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"] Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.671750 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.671930 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.671986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.672157 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q9n2b\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.673004 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.674561 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.674583 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lj2km\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.674583 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.674670 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.674794 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.674973 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.676482 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.676703 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5m2tj" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.685594 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5m2tj" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.739354 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-system-cni-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-ovn\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1451f3-2475-4614-b6a8-df41425e93b8-dbus\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197126bf-b934-4bb7-b1a4-d30338ffc52e-konnectivity-ca\") pod \"konnectivity-agent-wpqxt\" (UID: \"197126bf-b934-4bb7-b1a4-d30338ffc52e\") " pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-os-release\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.900308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjx9k\" (UniqueName: \"kubernetes.io/projected/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-kube-api-access-wjx9k\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-systemd\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-cni-netd\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-ovnkube-script-lib\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mf8\" (UniqueName: \"kubernetes.io/projected/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-kube-api-access-q8mf8\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-netns\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-kubernetes\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysctl-conf\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-lib-modules\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-cni-bin\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-conf-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4zr\" (UniqueName: \"kubernetes.io/projected/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-kube-api-access-qp4zr\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-kubelet\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-slash\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-cni-bin\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748834 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/063206ae-2dba-4fe8-9def-da6101746a51-ovn-node-metrics-cert\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.748975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-run\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-tuned\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjrp\" (UniqueName: \"kubernetes.io/projected/8d7c05a8-7911-461d-a206-57df68595e5e-kube-api-access-jxjrp\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-systemd-units\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197126bf-b934-4bb7-b1a4-d30338ffc52e-agent-certs\") pod \"konnectivity-agent-wpqxt\" (UID: \"197126bf-b934-4bb7-b1a4-d30338ffc52e\") " pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-sys-fs\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-system-cni-dir\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-systemd\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3f0165-edbd-404e-9ca9-ca0c82314028-host\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-etc-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.901992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-socket-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-device-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-kubelet\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wglm\" (UniqueName: \"kubernetes.io/projected/cc3f0165-edbd-404e-9ca9-ca0c82314028-kube-api-access-7wglm\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-host\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-var-lib-kubelet\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-node-log\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-ovnkube-config\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a61dae83-f0ff-46cd-86f4-4a898c056602-cni-binary-copy\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-daemon-config\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8cn\" (UniqueName: \"kubernetes.io/projected/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-kube-api-access-wb8cn\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-socket-dir-parent\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-cni-multus\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97nq\" (UniqueName: \"kubernetes.io/projected/a61dae83-f0ff-46cd-86f4-4a898c056602-kube-api-access-p97nq\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3f0165-edbd-404e-9ca9-ca0c82314028-serviceca\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.902638 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-registration-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-iptables-alerter-script\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysconfig\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-sys\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-var-lib-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1451f3-2475-4614-b6a8-df41425e93b8-kubelet-config\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.749931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cnibin\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-k8s-cni-cncf-io\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-hostroot\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-multus-certs\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-modprobe-d\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbvm\" (UniqueName: \"kubernetes.io/projected/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-kube-api-access-qqbvm\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-etc-kubernetes\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.903293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-cni-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-cnibin\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-host-slash\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysctl-d\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750426 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-tmp\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-tmp-dir\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-run-netns\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-log-socket\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-hosts-file\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-env-overrides\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwvn\" (UniqueName: \"kubernetes.io/projected/063206ae-2dba-4fe8-9def-da6101746a51-kube-api-access-hnwvn\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750597 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-os-release\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.750622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.850859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-cni-bin\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.850899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/063206ae-2dba-4fe8-9def-da6101746a51-ovn-node-metrics-cert\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.850943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.904018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.850946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-cni-bin\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.850968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.850988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-run\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-tuned\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjrp\" (UniqueName: \"kubernetes.io/projected/8d7c05a8-7911-461d-a206-57df68595e5e-kube-api-access-jxjrp\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.851083 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-systemd-units\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851109 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-run\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197126bf-b934-4bb7-b1a4-d30338ffc52e-agent-certs\") pod \"konnectivity-agent-wpqxt\" (UID: \"197126bf-b934-4bb7-b1a4-d30338ffc52e\") " pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.851194 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:24.351146062 +0000 UTC m=+2.149366598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-sys-fs\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-system-cni-dir\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-system-cni-dir\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851301 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-sys-fs\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-systemd\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-systemd-units\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.904683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3f0165-edbd-404e-9ca9-ca0c82314028-host\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-systemd\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-etc-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-socket-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851467 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3f0165-edbd-404e-9ca9-ca0c82314028-host\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-device-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-etc-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-kubelet\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-kubelet\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wglm\" (UniqueName: \"kubernetes.io/projected/cc3f0165-edbd-404e-9ca9-ca0c82314028-kube-api-access-7wglm\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-socket-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-host\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-var-lib-kubelet\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851625 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-host\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-node-log\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-ovnkube-config\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.905337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-device-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a61dae83-f0ff-46cd-86f4-4a898c056602-cni-binary-copy\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-var-lib-kubelet\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-daemon-config\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8cn\" (UniqueName: \"kubernetes.io/projected/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-kube-api-access-wb8cn\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-socket-dir-parent\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-cni-multus\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97nq\" (UniqueName: \"kubernetes.io/projected/a61dae83-f0ff-46cd-86f4-4a898c056602-kube-api-access-p97nq\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3f0165-edbd-404e-9ca9-ca0c82314028-serviceca\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-registration-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-iptables-alerter-script\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysconfig\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-sys\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852049 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-var-lib-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1451f3-2475-4614-b6a8-df41425e93b8-kubelet-config\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.906044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cnibin\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-k8s-cni-cncf-io\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-hostroot\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-multus-certs\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-modprobe-d\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbvm\" (UniqueName: \"kubernetes.io/projected/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-kube-api-access-qqbvm\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-etc-kubernetes\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a61dae83-f0ff-46cd-86f4-4a898c056602-cni-binary-copy\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-cni-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-cnibin\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-host-slash\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysctl-d\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-daemon-config\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-tmp\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.906681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-ovnkube-config\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-tmp-dir\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cnibin\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-run-netns\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-log-socket\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-registration-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-hosts-file\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-env-overrides\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvn\" (UniqueName: \"kubernetes.io/projected/063206ae-2dba-4fe8-9def-da6101746a51-kube-api-access-hnwvn\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-os-release\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852882 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-var-lib-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852941 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-sys\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852967 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysconfig\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.851794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-node-log\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.852988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1451f3-2475-4614-b6a8-df41425e93b8-kubelet-config\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-cni-multus\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-socket-dir-parent\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-k8s-cni-cncf-io\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853144 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-multus-certs\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-modprobe-d\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853332 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-hostroot\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853376 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-iptables-alerter-script\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-etc-kubernetes\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853437 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc3f0165-edbd-404e-9ca9-ca0c82314028-serviceca\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-host-slash\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853463 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-run-netns\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-cnibin\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-cni-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-os-release\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853685 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysctl-d\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-hosts-file\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.907897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853809 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.853860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-log-socket\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854014 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-tmp-dir\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-system-cni-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-env-overrides\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-system-cni-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-ovn\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854238 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1451f3-2475-4614-b6a8-df41425e93b8-dbus\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197126bf-b934-4bb7-b1a4-d30338ffc52e-konnectivity-ca\") pod \"konnectivity-agent-wpqxt\" (UID: \"197126bf-b934-4bb7-b1a4-d30338ffc52e\") " pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-os-release\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjx9k\" (UniqueName: \"kubernetes.io/projected/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-kube-api-access-wjx9k\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1451f3-2475-4614-b6a8-df41425e93b8-dbus\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-os-release\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-ovn\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-systemd\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.908651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-cni-netd\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-ovnkube-script-lib\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-systemd\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mf8\" (UniqueName: \"kubernetes.io/projected/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-kube-api-access-q8mf8\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-cni-netd\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.854547 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.854703 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:24.354685168 +0000 UTC m=+2.152905689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-netns\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-kubernetes\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysctl-conf\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-lib-modules\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-run-netns\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-kubernetes\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-cni-bin\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854921 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-conf-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.909293 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4zr\" (UniqueName: \"kubernetes.io/projected/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-kube-api-access-qp4zr\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-kubelet\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-host-var-lib-cni-bin\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-slash\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a61dae83-f0ff-46cd-86f4-4a898c056602-multus-conf-dir\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/197126bf-b934-4bb7-b1a4-d30338ffc52e-konnectivity-ca\") pod \"konnectivity-agent-wpqxt\" (UID: \"197126bf-b934-4bb7-b1a4-d30338ffc52e\") " pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855113 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-tuned\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.854966 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-etc-sysctl-conf\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-kubelet\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-run-openvswitch\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/063206ae-2dba-4fe8-9def-da6101746a51-host-slash\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855237 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-lib-modules\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/063206ae-2dba-4fe8-9def-da6101746a51-ovn-node-metrics-cert\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855824 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/063206ae-2dba-4fe8-9def-da6101746a51-ovnkube-script-lib\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855837 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-tmp\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.855847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/197126bf-b934-4bb7-b1a4-d30338ffc52e-agent-certs\") pod \"konnectivity-agent-wpqxt\" (UID: \"197126bf-b934-4bb7-b1a4-d30338ffc52e\") " pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.859821 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:23.909823 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.859841 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.859854 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:23.859935 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:24.359917809 +0000 UTC m=+2.158138332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.865766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8cn\" (UniqueName: \"kubernetes.io/projected/25c4651e-3a67-4a78-a4b0-57598fb5f5b0-kube-api-access-wb8cn\") pod \"node-resolver-d6m5b\" (UID: \"25c4651e-3a67-4a78-a4b0-57598fb5f5b0\") " pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.865834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wglm\" (UniqueName: \"kubernetes.io/projected/cc3f0165-edbd-404e-9ca9-ca0c82314028-kube-api-access-7wglm\") pod \"node-ca-xbgpm\" (UID: \"cc3f0165-edbd-404e-9ca9-ca0c82314028\") " pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.865965 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjrp\" (UniqueName: \"kubernetes.io/projected/8d7c05a8-7911-461d-a206-57df68595e5e-kube-api-access-jxjrp\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.868471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4zr\" (UniqueName: \"kubernetes.io/projected/64a96c3d-955b-4626-bbd8-0aaa61f8b0cb-kube-api-access-qp4zr\") pod \"iptables-alerter-4rzl5\" (UID: \"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb\") " pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.868682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjx9k\" (UniqueName: \"kubernetes.io/projected/5dd48287-5c6a-4c89-be63-8a1f0600fa8e-kube-api-access-wjx9k\") pod \"tuned-8wl75\" (UID: \"5dd48287-5c6a-4c89-be63-8a1f0600fa8e\") " pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.871068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97nq\" (UniqueName: \"kubernetes.io/projected/a61dae83-f0ff-46cd-86f4-4a898c056602-kube-api-access-p97nq\") pod \"multus-k8xpd\" (UID: \"a61dae83-f0ff-46cd-86f4-4a898c056602\") " pod="openshift-multus/multus-k8xpd" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.871091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mf8\" (UniqueName: \"kubernetes.io/projected/58bdbaff-323b-49e4-a33c-3ba51e4ed9a6-kube-api-access-q8mf8\") pod \"multus-additional-cni-plugins-hrjzl\" (UID: \"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6\") " pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.871094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbvm\" (UniqueName: \"kubernetes.io/projected/c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69-kube-api-access-qqbvm\") pod \"aws-ebs-csi-driver-node-tcwkq\" (UID: \"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:23.910518 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.871443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwvn\" (UniqueName: \"kubernetes.io/projected/063206ae-2dba-4fe8-9def-da6101746a51-kube-api-access-hnwvn\") pod \"ovnkube-node-s8c2d\" (UID: \"063206ae-2dba-4fe8-9def-da6101746a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:23.936244 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09061521089a77af38194f9d3efcc858.slice/crio-8e5927cc17daf20a116de108ddb916703f58befd64d861fadd098aff61299173 WatchSource:0}: Error finding container 8e5927cc17daf20a116de108ddb916703f58befd64d861fadd098aff61299173: Status 404 returned error can't find the container with id 8e5927cc17daf20a116de108ddb916703f58befd64d861fadd098aff61299173 Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:23.936458 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa83e0abbce66175145d3331b72251e.slice/crio-af081ca40f1178fecb5c3829801407f55519b11b19234e598d7a8217a9a37d51 WatchSource:0}: Error finding container af081ca40f1178fecb5c3829801407f55519b11b19234e598d7a8217a9a37d51: Status 404 returned error can't find the container with id af081ca40f1178fecb5c3829801407f55519b11b19234e598d7a8217a9a37d51 Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.943435 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.975986 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:23.985497 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197126bf_b934_4bb7_b1a4_d30338ffc52e.slice/crio-06b42e0743854f964016c3efd288cc70569bddb6a3ab0324d7418f3484ef8cd1 WatchSource:0}: Error finding container 06b42e0743854f964016c3efd288cc70569bddb6a3ab0324d7418f3484ef8cd1: Status 404 returned error can't find the container with id 06b42e0743854f964016c3efd288cc70569bddb6a3ab0324d7418f3484ef8cd1 Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:23.990436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:23.998439 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c8f6a7_dc47_4d17_97e1_ccf8282d0b69.slice/crio-391f1debcf32bdb40bc804e8f4eb493f03e4b677c2189ce910ee1b7876ee2c58 WatchSource:0}: Error finding container 391f1debcf32bdb40bc804e8f4eb493f03e4b677c2189ce910ee1b7876ee2c58: Status 404 returned error can't find the container with id 391f1debcf32bdb40bc804e8f4eb493f03e4b677c2189ce910ee1b7876ee2c58 Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.005021 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d6m5b" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.013447 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c4651e_3a67_4a78_a4b0_57598fb5f5b0.slice/crio-df2d6150b33ebccf559a79af7c1e02c5c3d6edc31489f7b311c1cf4f16e36dfb WatchSource:0}: Error finding container df2d6150b33ebccf559a79af7c1e02c5c3d6edc31489f7b311c1cf4f16e36dfb: Status 404 returned error can't find the container with id df2d6150b33ebccf559a79af7c1e02c5c3d6edc31489f7b311c1cf4f16e36dfb Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.014031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8wl75" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.023384 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd48287_5c6a_4c89_be63_8a1f0600fa8e.slice/crio-04d32ed94aeee16dd465465464b523c74feb1d7c635c38e7b05659449f1f996e WatchSource:0}: Error finding container 04d32ed94aeee16dd465465464b523c74feb1d7c635c38e7b05659449f1f996e: Status 404 returned error can't find the container with id 04d32ed94aeee16dd465465464b523c74feb1d7c635c38e7b05659449f1f996e Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.033562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xbgpm" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.042119 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3f0165_edbd_404e_9ca9_ca0c82314028.slice/crio-f7c0123fd35b708a0642f368ab1aaf5d081153f2d668169deb6a61ae5111a3fc WatchSource:0}: Error finding container f7c0123fd35b708a0642f368ab1aaf5d081153f2d668169deb6a61ae5111a3fc: Status 404 returned error can't find the container with id f7c0123fd35b708a0642f368ab1aaf5d081153f2d668169deb6a61ae5111a3fc Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.047097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.055881 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bdbaff_323b_49e4_a33c_3ba51e4ed9a6.slice/crio-b8745161e9b1e07294b60c5ab2a76fd20097c673beb8e983350218ed542e7fcc WatchSource:0}: Error finding container b8745161e9b1e07294b60c5ab2a76fd20097c673beb8e983350218ed542e7fcc: Status 404 returned error can't find the container with id b8745161e9b1e07294b60c5ab2a76fd20097c673beb8e983350218ed542e7fcc Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.064062 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k8xpd" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.070083 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4rzl5" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.074978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:15:24.139765 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.077615 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61dae83_f0ff_46cd_86f4_4a898c056602.slice/crio-4b1a484cb4bd663f56bfe5bc475c7293acae28f91cd8c46be08bb311a14d0651 WatchSource:0}: Error finding container 4b1a484cb4bd663f56bfe5bc475c7293acae28f91cd8c46be08bb311a14d0651: Status 404 returned error can't find the container with id 4b1a484cb4bd663f56bfe5bc475c7293acae28f91cd8c46be08bb311a14d0651 Jul 09 16:15:24.142626 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.080979 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a96c3d_955b_4626_bbd8_0aaa61f8b0cb.slice/crio-5937a450eaa2c1df25e5fe8c1ce1a0efee809c5fec5ba65eda2582b8b9123f20 WatchSource:0}: Error finding container 5937a450eaa2c1df25e5fe8c1ce1a0efee809c5fec5ba65eda2582b8b9123f20: Status 404 returned error can't find the container with id 5937a450eaa2c1df25e5fe8c1ce1a0efee809c5fec5ba65eda2582b8b9123f20 Jul 09 16:15:24.142626 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:15:24.085213 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063206ae_2dba_4fe8_9def_da6101746a51.slice/crio-9aa55f845dede9432d9702f61ed15e268490c53f9c888c21916f8fea43bf0129 WatchSource:0}: Error finding container 9aa55f845dede9432d9702f61ed15e268490c53f9c888c21916f8fea43bf0129: Status 404 returned error can't find the container with id 9aa55f845dede9432d9702f61ed15e268490c53f9c888c21916f8fea43bf0129 Jul 09 16:15:24.142626 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.089550 2562 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-s8c2d_openshift-ovn-kubernetes(063206ae-2dba-4fe8-9def-da6101746a51): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 09 16:15:24.142626 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.090803 2562 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-s8c2d_openshift-ovn-kubernetes(063206ae-2dba-4fe8-9def-da6101746a51): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 09 16:15:24.142626 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.094167 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" podUID="063206ae-2dba-4fe8-9def-da6101746a51" Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.361137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.361180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.361201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361301 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361309 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361320 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361327 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361338 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361366 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:25.361351226 +0000 UTC m=+3.159571750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361386 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:25.361379165 +0000 UTC m=+3.159599685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:24.436219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.361400 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:25.361393473 +0000 UTC m=+3.159613994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:24.437121 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.436932 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:24.758789 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.690179 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:10:24 +0000 UTC" deadline="2028-02-17 17:27:12.987840461 +0000 UTC" Jul 09 16:15:24.758789 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.690558 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14113h11m48.297285884s" Jul 09 16:15:24.758789 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.719047 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:24.852167 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.818328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:24.852167 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.818465 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:24.852167 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.828826 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"9aa55f845dede9432d9702f61ed15e268490c53f9c888c21916f8fea43bf0129"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.880731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4rzl5" event={"ID":"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb","Type":"ContainerStarted","Data":"5937a450eaa2c1df25e5fe8c1ce1a0efee809c5fec5ba65eda2582b8b9123f20"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:24.890540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" podUID="063206ae-2dba-4fe8-9def-da6101746a51" Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.894006 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerStarted","Data":"b8745161e9b1e07294b60c5ab2a76fd20097c673beb8e983350218ed542e7fcc"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.906005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xbgpm" event={"ID":"cc3f0165-edbd-404e-9ca9-ca0c82314028","Type":"ContainerStarted","Data":"f7c0123fd35b708a0642f368ab1aaf5d081153f2d668169deb6a61ae5111a3fc"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.910917 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d6m5b" event={"ID":"25c4651e-3a67-4a78-a4b0-57598fb5f5b0","Type":"ContainerStarted","Data":"df2d6150b33ebccf559a79af7c1e02c5c3d6edc31489f7b311c1cf4f16e36dfb"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.913957 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" event={"ID":"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69","Type":"ContainerStarted","Data":"391f1debcf32bdb40bc804e8f4eb493f03e4b677c2189ce910ee1b7876ee2c58"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.917647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wpqxt" event={"ID":"197126bf-b934-4bb7-b1a4-d30338ffc52e","Type":"ContainerStarted","Data":"06b42e0743854f964016c3efd288cc70569bddb6a3ab0324d7418f3484ef8cd1"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.920295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" event={"ID":"09061521089a77af38194f9d3efcc858","Type":"ContainerStarted","Data":"8e5927cc17daf20a116de108ddb916703f58befd64d861fadd098aff61299173"} Jul 09 16:15:24.928921 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.924166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k8xpd" event={"ID":"a61dae83-f0ff-46cd-86f4-4a898c056602","Type":"ContainerStarted","Data":"4b1a484cb4bd663f56bfe5bc475c7293acae28f91cd8c46be08bb311a14d0651"} Jul 09 16:15:24.932929 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.932191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8wl75" event={"ID":"5dd48287-5c6a-4c89-be63-8a1f0600fa8e","Type":"ContainerStarted","Data":"04d32ed94aeee16dd465465464b523c74feb1d7c635c38e7b05659449f1f996e"} Jul 09 16:15:24.937120 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:24.937079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"0aa83e0abbce66175145d3331b72251e","Type":"ContainerStarted","Data":"af081ca40f1178fecb5c3829801407f55519b11b19234e598d7a8217a9a37d51"} Jul 09 16:15:25.368871 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.368818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:25.369071 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.368881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:25.369071 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.368947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:25.369212 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369100 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:25.369212 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369168 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:27.369147408 +0000 UTC m=+5.167367945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:25.369331 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369252 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:25.369331 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369287 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:27.369276301 +0000 UTC m=+5.167496837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:25.369426 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369383 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:25.369426 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369403 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:25.369426 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369417 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:25.369568 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.369455 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:27.369443362 +0000 UTC m=+5.167663885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:25.691520 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.691389 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:10:24 +0000 UTC" deadline="2028-01-09 09:44:51.007233418 +0000 UTC" Jul 09 16:15:25.691520 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.691455 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13169h29m25.315783516s" Jul 09 16:15:25.814400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.814361 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:25.814563 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.814510 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:25.814973 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.814929 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:25.815088 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:25.815048 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:25.858060 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:25.857817 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:26.815242 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:26.815166 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:26.815725 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:26.815319 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:27.393431 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:27.393389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:27.393621 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:27.393451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:27.393621 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:27.393489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:27.393726 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393661 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:27.393726 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393682 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:27.393726 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393707 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:27.393877 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393770 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:31.393750999 +0000 UTC m=+9.191971541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:27.393877 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393853 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:27.394016 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393890 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:31.393878351 +0000 UTC m=+9.192098873 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:27.394016 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.393966 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:27.394139 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.394016 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:31.394001079 +0000 UTC m=+9.192221618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:27.814906 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:27.814793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:27.814906 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:27.814854 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:27.815116 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.814992 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:27.815171 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:27.815136 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:28.814427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:28.814355 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:28.814904 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:28.814515 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:29.814394 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:29.814361 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:29.814580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:29.814399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:29.814580 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:29.814501 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:29.815058 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:29.814610 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:30.814189 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:30.813893 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:30.814380 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:30.814320 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:31.427137 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:31.427103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:31.427152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:31.427200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427287 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427323 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427339 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427351 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:39.427330328 +0000 UTC m=+17.225550855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427352 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427366 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427392 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:39.427382191 +0000 UTC m=+17.225602716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:31.427570 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.427431 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:39.427409962 +0000 UTC m=+17.225630484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:31.814701 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:31.814663 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:31.814701 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:31.814705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:31.814927 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.814772 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:31.814927 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:31.814844 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:32.814838 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:32.814801 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:32.815291 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:32.814930 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:33.814133 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.814046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:33.814262 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:33.814160 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:33.814262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.814211 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:33.814330 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:33.814304 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:33.958608 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.958574 2562 generic.go:356] "Generic (PLEG): container finished" podID="58bdbaff-323b-49e4-a33c-3ba51e4ed9a6" containerID="3b7c24bcef05991bf2cc6366a6f98adbde88c7a4c57c268f0bbcc0520be4c4c5" exitCode=0 Jul 09 16:15:33.959100 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.958650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerDied","Data":"3b7c24bcef05991bf2cc6366a6f98adbde88c7a4c57c268f0bbcc0520be4c4c5"} Jul 09 16:15:33.960241 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.960222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xbgpm" event={"ID":"cc3f0165-edbd-404e-9ca9-ca0c82314028","Type":"ContainerStarted","Data":"24b9b1bf5eb0ece2b383d5f7e56ddb048dc54f976ebd75a793d2992402a66c79"} Jul 09 16:15:33.961786 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.961758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d6m5b" event={"ID":"25c4651e-3a67-4a78-a4b0-57598fb5f5b0","Type":"ContainerStarted","Data":"9f54a8dd0c89fc963a4f3cc7ae0fe56301473f506fb43a16825f9a70a03eade5"} Jul 09 16:15:33.963153 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.963132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" event={"ID":"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69","Type":"ContainerStarted","Data":"a05b47f5030478adb99b594d6420ebffe45b3153927c4f32e0b7223aacf1792d"} Jul 09 16:15:33.964471 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.964448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wpqxt" event={"ID":"197126bf-b934-4bb7-b1a4-d30338ffc52e","Type":"ContainerStarted","Data":"97ec17df8f62d925452f00071d50ca675ae1ed1f7be15063e503ca19f63ec0e6"} Jul 09 16:15:33.968797 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.968768 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" event={"ID":"09061521089a77af38194f9d3efcc858","Type":"ContainerStarted","Data":"e28e0f35d83174d07720e60578981c512e62121831576e6d69108504d8b14178"} Jul 09 16:15:33.970086 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.970065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8wl75" event={"ID":"5dd48287-5c6a-4c89-be63-8a1f0600fa8e","Type":"ContainerStarted","Data":"60a13bf1620703935a53584865add48691c34f5935df76599a11876e156f7446"} Jul 09 16:15:33.971402 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.971382 2562 generic.go:356] "Generic (PLEG): container finished" podID="0aa83e0abbce66175145d3331b72251e" containerID="d6b914cc3d51279256a9cf42155b12ee5b4c47895c42033e1b7a184d0297f896" exitCode=0 Jul 09 16:15:33.971465 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:33.971411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"0aa83e0abbce66175145d3331b72251e","Type":"ContainerDied","Data":"d6b914cc3d51279256a9cf42155b12ee5b4c47895c42033e1b7a184d0297f896"} Jul 09 16:15:34.002791 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.002734 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wpqxt" podStartSLOduration=1.7009115110000002 podStartE2EDuration="11.002715228s" totalImagesPullingTime="9.301803717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:34.002198622 +0000 UTC m=+11.800419162" watchObservedRunningTime="2026-07-09 16:15:34.002715228 +0000 UTC m=+11.800935770" Jul 09 16:15:34.032106 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.031930 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xbgpm" podStartSLOduration=1.781571612 podStartE2EDuration="11.031914279s" totalImagesPullingTime="9.250342667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:34.017036546 +0000 UTC m=+11.815257087" watchObservedRunningTime="2026-07-09 16:15:34.031914279 +0000 UTC m=+11.830134813" Jul 09 16:15:34.049598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.049557 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" podStartSLOduration=10.049541676 podStartE2EDuration="10.049541676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:34.049190306 +0000 UTC m=+11.847410847" watchObservedRunningTime="2026-07-09 16:15:34.049541676 +0000 UTC m=+11.847762216" Jul 09 16:15:34.070022 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.069935 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d6m5b" podStartSLOduration=1.789086723 podStartE2EDuration="11.069863135s" totalImagesPullingTime="9.280776412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:34.069715854 +0000 UTC m=+11.867936394" watchObservedRunningTime="2026-07-09 16:15:34.069863135 +0000 UTC m=+11.868083674" Jul 09 16:15:34.087114 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.087065 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8wl75" podStartSLOduration=1.771795526 podStartE2EDuration="11.087046289s" totalImagesPullingTime="9.315250763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:34.086946774 +0000 UTC m=+11.885167314" watchObservedRunningTime="2026-07-09 16:15:34.087046289 +0000 UTC m=+11.885266881" Jul 09 16:15:34.814195 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.814154 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:34.814370 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:34.814321 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:34.975570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.975536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"0aa83e0abbce66175145d3331b72251e","Type":"ContainerStarted","Data":"e485e0445c48172e8c2efb78b342a19fa9306da4282ad4005134df2b7680142f"} Jul 09 16:15:34.977222 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.977195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4rzl5" event={"ID":"64a96c3d-955b-4626-bbd8-0aaa61f8b0cb","Type":"ContainerStarted","Data":"496b44ae274d751782a76adfcc2a013c6a4098ffd2a0dc8f5ee7e2af75b81796"} Jul 09 16:15:34.989639 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:34.989584 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" podStartSLOduration=10.98956573 podStartE2EDuration="10.98956573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:34.989084582 +0000 UTC m=+12.787305123" watchObservedRunningTime="2026-07-09 16:15:34.98956573 +0000 UTC m=+12.787786270" Jul 09 16:15:35.005394 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.005338 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4rzl5" podStartSLOduration=2.792341527 podStartE2EDuration="12.005326819s" totalImagesPullingTime="9.212985292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:35.004816011 +0000 UTC m=+12.803036542" watchObservedRunningTime="2026-07-09 16:15:35.005326819 +0000 UTC m=+12.803547359" Jul 09 16:15:35.116978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.116942 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 16:15:35.707988 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.707862 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T16:15:35.116966002Z","UUID":"ca6e6519-06f4-4cbd-be8c-65fa785132b4","Handler":null,"Name":"","Endpoint":""} Jul 09 16:15:35.710462 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.710431 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 16:15:35.710576 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.710480 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 16:15:35.814857 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.814830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:35.815044 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:35.814963 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:35.815398 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.815376 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:35.815495 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:35.815473 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:35.980698 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:35.980613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" event={"ID":"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69","Type":"ContainerStarted","Data":"8fddadfdb5eaa289121d725abb06f412cfc058bd3e0f8f91979dac8533759d29"} Jul 09 16:15:36.814301 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:36.814271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:36.814471 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:36.814401 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:37.814128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:37.814093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:37.814128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:37.814123 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:37.814757 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:37.814219 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:37.814757 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:37.814354 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:38.588916 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:38.588867 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:38.589667 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:38.589646 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:38.814629 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:38.814591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:38.814983 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:38.814751 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:39.496369 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:39.496339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:39.496567 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:39.496397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:39.496567 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:39.496436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:39.496567 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496503 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:39.496567 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496511 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:39.496567 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496526 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:39.496567 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496537 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:39.496774 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496571 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:39.496774 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496577 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.496558957 +0000 UTC m=+33.294779496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:39.496774 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496649 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.49663154 +0000 UTC m=+33.294852072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:39.496774 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.496671 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.496662073 +0000 UTC m=+33.294882614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:39.814403 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:39.814320 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:39.814403 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:39.814335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:39.814600 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.814444 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:39.814600 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:39.814556 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:40.814220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:40.814187 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:40.814408 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:40.814371 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:41.372697 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:41.372650 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:41.373135 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:41.372762 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 16:15:41.373323 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:41.373291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wpqxt" Jul 09 16:15:41.814062 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:41.813989 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:41.814227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:41.813990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:41.814227 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:41.814096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:41.814227 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:41.814193 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:42.815675 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:42.815464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:42.816136 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:42.815771 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:42.996463 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:42.996424 2562 generic.go:356] "Generic (PLEG): container finished" podID="58bdbaff-323b-49e4-a33c-3ba51e4ed9a6" containerID="e7e9b50a632f3d98bf37435e7095f0749665180c88cf336bd9cd69b803173d90" exitCode=0 Jul 09 16:15:42.996624 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:42.996491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerDied","Data":"e7e9b50a632f3d98bf37435e7095f0749665180c88cf336bd9cd69b803173d90"} Jul 09 16:15:42.998557 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:42.998534 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" event={"ID":"c1c8f6a7-dc47-4d17-97e1-ccf8282d0b69","Type":"ContainerStarted","Data":"43a889fa9ca758502e58334a462d9ed3d24b8e58f16337eb5798dee9c97e34b6"} Jul 09 16:15:42.999904 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:42.999867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k8xpd" event={"ID":"a61dae83-f0ff-46cd-86f4-4a898c056602","Type":"ContainerStarted","Data":"487ab849010e6938f148a7f3cb90a1c5295eb10bbbfb87620bc724891a7affb6"} Jul 09 16:15:43.040477 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:43.040423 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k8xpd" podStartSLOduration=2.104717318 podStartE2EDuration="20.040406907s" totalImagesPullingTime="17.935689589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:43.039860545 +0000 UTC m=+20.838081088" watchObservedRunningTime="2026-07-09 16:15:43.040406907 +0000 UTC m=+20.838627452" Jul 09 16:15:43.058201 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:43.058162 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tcwkq" podStartSLOduration=2.541108959 podStartE2EDuration="20.058148683s" totalImagesPullingTime="17.517039724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:43.058011528 +0000 UTC m=+20.856232070" watchObservedRunningTime="2026-07-09 16:15:43.058148683 +0000 UTC m=+20.856369223" Jul 09 16:15:43.814627 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:43.814571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:43.814857 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:43.814703 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:43.814978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:43.814578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:43.815098 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:43.815070 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:44.814697 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:44.814662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:44.815238 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:44.814800 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:45.006449 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:45.006420 2562 generic.go:356] "Generic (PLEG): container finished" podID="58bdbaff-323b-49e4-a33c-3ba51e4ed9a6" containerID="d3c187d5d964cfa1ac6cc717390f50d7491aace8346d647e875dcdd12109c121" exitCode=0 Jul 09 16:15:45.006614 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:45.006466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerDied","Data":"d3c187d5d964cfa1ac6cc717390f50d7491aace8346d647e875dcdd12109c121"} Jul 09 16:15:45.814769 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:45.814742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:45.815106 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:45.814742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:45.815106 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:45.814866 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:45.815106 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:45.814956 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:46.010340 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:46.010313 2562 generic.go:356] "Generic (PLEG): container finished" podID="58bdbaff-323b-49e4-a33c-3ba51e4ed9a6" containerID="d3d50726aa9b6b76710b5e8847395d1c369c179e69c59e5858fddee16552daa6" exitCode=0 Jul 09 16:15:46.010450 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:46.010349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerDied","Data":"d3d50726aa9b6b76710b5e8847395d1c369c179e69c59e5858fddee16552daa6"} Jul 09 16:15:46.814945 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:46.814911 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:46.815404 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:46.815042 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:47.814091 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:47.814049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:47.814273 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:47.814049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:47.814273 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:47.814188 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:47.814273 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:47.814233 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:48.814859 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:48.814768 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:48.815305 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:48.814920 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:49.813957 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:49.813925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:49.814233 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:49.813928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:49.814233 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:49.814045 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:49.814233 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:49.814117 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:50.813977 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:50.813933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:50.814443 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:50.814169 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:51.814004 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:51.813925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:51.814334 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:51.813925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:51.814334 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:51.814041 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:51.814334 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:51.814093 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:52.025274 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:52.025236 2562 generic.go:356] "Generic (PLEG): container finished" podID="58bdbaff-323b-49e4-a33c-3ba51e4ed9a6" containerID="3bcc3d9254921ed46b41aad549f1a8703c70612e158bf39bcec3206b2f6e3a51" exitCode=0 Jul 09 16:15:52.025425 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:52.025298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerDied","Data":"3bcc3d9254921ed46b41aad549f1a8703c70612e158bf39bcec3206b2f6e3a51"} Jul 09 16:15:52.815145 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:52.815112 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:52.815546 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:52.815236 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:53.031098 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:53.031060 2562 generic.go:356] "Generic (PLEG): container finished" podID="58bdbaff-323b-49e4-a33c-3ba51e4ed9a6" containerID="f0dc36f863acc828577e436e514dcf56ce59513f77f6eddf8cff494d8074ab4d" exitCode=0 Jul 09 16:15:53.031249 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:53.031120 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerDied","Data":"f0dc36f863acc828577e436e514dcf56ce59513f77f6eddf8cff494d8074ab4d"} Jul 09 16:15:53.814061 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:53.814029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:53.814220 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:53.814130 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:53.814220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:53.814186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:53.814382 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:53.814269 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:54.035994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:54.035960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" event={"ID":"58bdbaff-323b-49e4-a33c-3ba51e4ed9a6","Type":"ContainerStarted","Data":"14d12fcdce05a8dc8e612238755e777b578e2721a12af395ab74a39e47e04d45"} Jul 09 16:15:54.063975 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:54.063927 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hrjzl" podStartSLOduration=6.371019303 podStartE2EDuration="31.063913652s" totalImagesPullingTime="24.692894349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:54.062342575 +0000 UTC m=+31.860563115" watchObservedRunningTime="2026-07-09 16:15:54.063913652 +0000 UTC m=+31.862134193" Jul 09 16:15:54.814706 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:54.814674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:54.814850 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:54.814811 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:55.518891 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:55.518847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:55.518922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:55.518960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519004 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519074 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519090 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:16:27.519074038 +0000 UTC m=+65.317294558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519095 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519115 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519119 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:27.519107343 +0000 UTC m=+65.317327863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519126 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:55.519354 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.519181 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:27.519164718 +0000 UTC m=+65.317385239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:55.814058 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:55.813978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:55.814209 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.814094 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:55.814209 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:55.814157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:55.814319 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:55.814266 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:56.813997 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:56.813963 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:56.814445 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:56.814096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:57.814133 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:57.814096 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:57.814597 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:57.814213 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:15:57.814597 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:57.814273 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:57.814597 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:57.814406 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:58.814764 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:58.814726 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:15:58.815246 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:58.814871 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:15:59.814247 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:59.814212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:15:59.814438 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:59.814330 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:15:59.814438 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:15:59.814369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:15:59.814533 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:15:59.814436 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:00.814606 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:00.814432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:00.815321 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:00.814697 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:01.048590 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.048494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wpqxt_197126bf-b934-4bb7-b1a4-d30338ffc52e/konnectivity-agent/0.log" limit=1024 Jul 09 16:16:01.051938 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.051918 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:16:01.052248 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052229 2562 generic.go:356] "Generic (PLEG): container finished" podID="063206ae-2dba-4fe8-9def-da6101746a51" containerID="8725d42830ab2a66d6113773d6c6f8cee9600303dc51a260e5102837048926cb" exitCode=1 Jul 09 16:16:01.052288 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"e53915cc9b7e27aedab6362e567fc41266569b5196ba71aae98966ae3cdd8fcb"} Jul 09 16:16:01.052338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"a651315187264bea5915d35398775ba8babc49be3159d1d8a616d961c0a30fd9"} Jul 09 16:16:01.052368 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"faa79573cee3c083732dc75cd7861b0e2da75a817b4a65ae8463362b47699b31"} Jul 09 16:16:01.052403 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"97589313dd5ba42ba32542b15d12d73613ea690050e22f024d3359b341a2da82"} Jul 09 16:16:01.052436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerDied","Data":"8725d42830ab2a66d6113773d6c6f8cee9600303dc51a260e5102837048926cb"} Jul 09 16:16:01.052464 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.052445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"f7f1e2375f5802fb4176edc9357d1f25c02931313f6add2f3eafd4187b5774f1"} Jul 09 16:16:01.132665 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.132622 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wpqxt_197126bf-b934-4bb7-b1a4-d30338ffc52e/konnectivity-agent/0.log" limit=1024 Jul 09 16:16:01.814512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.814484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:01.814694 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:01.814486 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:01.814694 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:01.814591 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:01.814694 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:01.814664 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:02.815403 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:02.815381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:02.815713 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:02.815466 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:03.060016 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:03.059943 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:16:03.060294 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:03.060270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"583ad4cf6b11ad071bcb41808d79959e9b1e14b12c2aa4024e51f83125317f5a"} Jul 09 16:16:03.814357 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:03.814318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:03.814546 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:03.814318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:03.814546 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:03.814421 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:03.814695 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:03.814580 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:04.814617 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:04.814586 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:04.815001 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:04.814717 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:05.069184 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.068978 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:16:05.069636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.069535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"8b5c40ecb23358b75dfeef1504b93a8f06cdbe572c23d89c9696786583e8030f"} Jul 09 16:16:05.070043 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.069850 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:16:05.070589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.070108 2562 scope.go:117] "RemoveContainer" containerID="8725d42830ab2a66d6113773d6c6f8cee9600303dc51a260e5102837048926cb" Jul 09 16:16:05.070589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.070125 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:16:05.070589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.070192 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:16:05.089768 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.089749 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:16:05.090152 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.090133 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:16:05.274744 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.274721 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d6m5b_25c4651e-3a67-4a78-a4b0-57598fb5f5b0/dns-node-resolver/0.log" Jul 09 16:16:05.814262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.814230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:05.814439 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:05.814230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:05.814439 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:05.814337 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:05.814439 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:05.814398 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:06.075438 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:06.075353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:16:06.075998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:06.075714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" event={"ID":"063206ae-2dba-4fe8-9def-da6101746a51","Type":"ContainerStarted","Data":"9b1687b814cf41abea5f8c66c8b03c28718208131a36e03eae235e0e553e1431"} Jul 09 16:16:06.105142 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:06.105094 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" podStartSLOduration=7.086406212 podStartE2EDuration="43.105080647s" totalImagesPullingTime="36.018674435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-09 16:16:06.102129965 +0000 UTC m=+43.900350509" watchObservedRunningTime="2026-07-09 16:16:06.105080647 +0000 UTC m=+43.903301188" Jul 09 16:16:06.474504 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:06.474476 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xbgpm_cc3f0165-edbd-404e-9ca9-ca0c82314028/node-ca/0.log" Jul 09 16:16:06.814852 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:06.814772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:06.815038 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:06.814902 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:07.235398 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:07.235367 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8dkn"] Jul 09 16:16:07.235829 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:07.235495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:07.235829 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:07.235632 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:07.248487 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:07.248455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpbbn"] Jul 09 16:16:07.248639 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:07.248573 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:07.248715 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:07.248692 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:07.249565 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:07.249539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xlhnm"] Jul 09 16:16:07.249657 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:07.249629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:07.249726 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:07.249711 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:08.814359 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:08.814327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:08.814760 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:08.814328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:08.814760 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:08.814449 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:08.814760 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:08.814328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:08.814760 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:08.814576 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:08.814760 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:08.814511 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:10.814838 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:10.814657 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:10.815276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:10.814654 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:10.815276 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:10.814934 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:10.815276 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:10.815035 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:10.815276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:10.814655 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:10.815276 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:10.815130 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:12.815129 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:12.815093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:12.815543 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:12.815191 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:12.815543 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:12.815272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:12.815543 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:12.815372 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:12.815543 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:12.815372 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:12.815543 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:12.815443 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:14.814415 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:14.814371 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:14.814795 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:14.814390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:14.814795 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:14.814498 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:14.814795 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:14.814615 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:14.814795 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:14.814390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:14.814795 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:14.814726 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:16.814306 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:16.814268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:16.814306 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:16.814306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:16.814763 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:16.814321 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:16.814763 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:16.814396 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:16.814763 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:16.814504 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:16.814763 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:16.814578 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:18.814034 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:18.813990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:18.814034 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:18.814014 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:18.814034 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:18.813996 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:18.814507 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:18.814126 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:18.814507 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:18.814159 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:18.814507 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:18.814259 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:20.814307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:20.814272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:20.814723 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:20.814271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:20.814723 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:20.814396 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:20.814723 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:20.814280 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:20.814723 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:20.814482 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:20.814723 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:20.814534 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:22.815427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:22.815395 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:22.815901 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:22.815464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:22.815901 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:22.815478 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:22.815901 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:22.815549 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:22.815901 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:22.815594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:22.815901 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:22.815644 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:24.813994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:24.813957 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:24.814420 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:24.814072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:24.814420 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:24.814075 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:24.814420 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:24.814183 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:24.814420 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:24.814209 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:24.814420 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:24.814260 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:26.814351 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:26.814309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:26.814760 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:26.814309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:26.814760 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:26.814422 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xlhnm" podUID="174ae44f-0ade-475f-b740-c6606d184c02" Jul 09 16:16:26.814760 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:26.814325 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:26.814760 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:26.814564 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpbbn" podUID="8d7c05a8-7911-461d-a206-57df68595e5e" Jul 09 16:16:26.814760 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:26.814629 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8dkn" podUID="8b1451f3-2475-4614-b6a8-df41425e93b8" Jul 09 16:16:27.468194 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.468167 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeReady" Jul 09 16:16:27.468373 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.468308 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 16:16:27.511725 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.511692 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f68bbf98-jfs56"] Jul 09 16:16:27.513418 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.513402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.516955 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.516926 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6v4p5\"" Jul 09 16:16:27.516955 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.516928 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 16:16:27.517095 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.516934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 16:16:27.517412 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.517399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 16:16:27.523263 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.523209 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 16:16:27.528730 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.528708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f68bbf98-jfs56"] Jul 09 16:16:27.531716 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.531690 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rx2z5"] Jul 09 16:16:27.533391 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.533375 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.537566 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537549 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 16:16:27.537695 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:27.537695 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:27.537695 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537672 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 16:16:27.537862 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:27.537862 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537849 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537864 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537869 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537913 2562 projected.go:196] Error preparing data for projected volume kube-api-access-27sds for pod openshift-network-diagnostics/network-check-target-xlhnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537940 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537959 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs podName:8d7c05a8-7911-461d-a206-57df68595e5e nodeName:}" failed. No retries permitted until 2026-07-09 16:17:31.537937253 +0000 UTC m=+129.336157773 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs") pod "network-metrics-daemon-mpbbn" (UID: "8d7c05a8-7911-461d-a206-57df68595e5e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.537961 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6z7vc\"" Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537981 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds podName:174ae44f-0ade-475f-b740-c6606d184c02 nodeName:}" failed. No retries permitted until 2026-07-09 16:17:31.53797018 +0000 UTC m=+129.336190703 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-27sds" (UniqueName: "kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds") pod "network-check-target-xlhnm" (UID: "174ae44f-0ade-475f-b740-c6606d184c02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:16:27.538003 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.537988 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:16:27.538316 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:27.538043 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret podName:8b1451f3-2475-4614-b6a8-df41425e93b8 nodeName:}" failed. No retries permitted until 2026-07-09 16:17:31.538025519 +0000 UTC m=+129.336246051 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret") pod "global-pull-secret-syncer-b8dkn" (UID: "8b1451f3-2475-4614-b6a8-df41425e93b8") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:16:27.538990 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.538965 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-clfl2"] Jul 09 16:16:27.540818 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.540799 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.544598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.544572 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 16:16:27.544714 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.544699 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w5fn\"" Jul 09 16:16:27.544917 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.544871 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 16:16:27.548092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.548068 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rx2z5"] Jul 09 16:16:27.554536 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.554508 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clfl2"] Jul 09 16:16:27.629121 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.629082 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z8bns"] Jul 09 16:16:27.631014 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.630989 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.637993 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.637965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4a854d5-5647-4ac3-9d40-bb1d99832b48-registry-certificates\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638118 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4a854d5-5647-4ac3-9d40-bb1d99832b48-trusted-ca\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638118 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-registry-tls\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638118 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26741143-5141-49b1-96a9-7f82db5af8a7-cert\") pod \"ingress-canary-rx2z5\" (UID: \"26741143-5141-49b1-96a9-7f82db5af8a7\") " pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.638287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45m7j\" (UniqueName: \"kubernetes.io/projected/26741143-5141-49b1-96a9-7f82db5af8a7-kube-api-access-45m7j\") pod \"ingress-canary-rx2z5\" (UID: \"26741143-5141-49b1-96a9-7f82db5af8a7\") " pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.638287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-metrics-tls\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.638287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638246 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 16:16:27.638287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4a854d5-5647-4ac3-9d40-bb1d99832b48-installation-pull-secrets\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bl69\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-kube-api-access-9bl69\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-bound-sa-token\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-config-volume\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638363 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zg2\" (UniqueName: \"kubernetes.io/projected/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-kube-api-access-s7zg2\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4a854d5-5647-4ac3-9d40-bb1d99832b48-image-registry-private-configuration\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4a854d5-5647-4ac3-9d40-bb1d99832b48-ca-trust-extracted\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.638530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638482 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 16:16:27.638875 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-tmp-dir\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.638875 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638566 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 16:16:27.638875 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.638662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wt\"" Jul 09 16:16:27.645992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.645964 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z8bns"] Jul 09 16:16:27.739612 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-tmp-dir\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.739612 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4a854d5-5647-4ac3-9d40-bb1d99832b48-registry-certificates\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.739612 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4a854d5-5647-4ac3-9d40-bb1d99832b48-trusted-ca\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.739918 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-registry-tls\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.739918 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26741143-5141-49b1-96a9-7f82db5af8a7-cert\") pod \"ingress-canary-rx2z5\" (UID: \"26741143-5141-49b1-96a9-7f82db5af8a7\") " pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.739918 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81464632-b149-4ecf-b185-c77649824390-crio-socket\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.739918 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81464632-b149-4ecf-b185-c77649824390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.739918 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45m7j\" (UniqueName: \"kubernetes.io/projected/26741143-5141-49b1-96a9-7f82db5af8a7-kube-api-access-45m7j\") pod \"ingress-canary-rx2z5\" (UID: \"26741143-5141-49b1-96a9-7f82db5af8a7\") " pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-tmp-dir\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-metrics-tls\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.739990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4a854d5-5647-4ac3-9d40-bb1d99832b48-installation-pull-secrets\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bl69\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-kube-api-access-9bl69\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740046 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81464632-b149-4ecf-b185-c77649824390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrspb\" (UniqueName: \"kubernetes.io/projected/81464632-b149-4ecf-b185-c77649824390-kube-api-access-vrspb\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81464632-b149-4ecf-b185-c77649824390-data-volume\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.740160 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-bound-sa-token\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740509 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-config-volume\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.740509 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zg2\" (UniqueName: \"kubernetes.io/projected/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-kube-api-access-s7zg2\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.740611 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4a854d5-5647-4ac3-9d40-bb1d99832b48-registry-certificates\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740611 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4a854d5-5647-4ac3-9d40-bb1d99832b48-image-registry-private-configuration\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740720 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4a854d5-5647-4ac3-9d40-bb1d99832b48-ca-trust-extracted\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740916 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4a854d5-5647-4ac3-9d40-bb1d99832b48-trusted-ca\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.740916 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-config-volume\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.741046 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.740912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4a854d5-5647-4ac3-9d40-bb1d99832b48-ca-trust-extracted\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.743573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.743549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-registry-tls\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.743682 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.743572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-metrics-tls\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.743682 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.743641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4a854d5-5647-4ac3-9d40-bb1d99832b48-installation-pull-secrets\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.743788 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.743772 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4a854d5-5647-4ac3-9d40-bb1d99832b48-image-registry-private-configuration\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.743824 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.743774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26741143-5141-49b1-96a9-7f82db5af8a7-cert\") pod \"ingress-canary-rx2z5\" (UID: \"26741143-5141-49b1-96a9-7f82db5af8a7\") " pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.749020 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.748994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45m7j\" (UniqueName: \"kubernetes.io/projected/26741143-5141-49b1-96a9-7f82db5af8a7-kube-api-access-45m7j\") pod \"ingress-canary-rx2z5\" (UID: \"26741143-5141-49b1-96a9-7f82db5af8a7\") " pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.749198 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.749181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zg2\" (UniqueName: \"kubernetes.io/projected/9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb-kube-api-access-s7zg2\") pod \"dns-default-clfl2\" (UID: \"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb\") " pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.749238 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.749203 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bl69\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-kube-api-access-9bl69\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.749357 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.749342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a854d5-5647-4ac3-9d40-bb1d99832b48-bound-sa-token\") pod \"image-registry-5f68bbf98-jfs56\" (UID: \"d4a854d5-5647-4ac3-9d40-bb1d99832b48\") " pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.823128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.823081 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:27.841015 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.840978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81464632-b149-4ecf-b185-c77649824390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841147 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrspb\" (UniqueName: \"kubernetes.io/projected/81464632-b149-4ecf-b185-c77649824390-kube-api-access-vrspb\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841147 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81464632-b149-4ecf-b185-c77649824390-data-volume\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841223 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81464632-b149-4ecf-b185-c77649824390-crio-socket\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841223 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81464632-b149-4ecf-b185-c77649824390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841345 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81464632-b149-4ecf-b185-c77649824390-crio-socket\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841421 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81464632-b149-4ecf-b185-c77649824390-data-volume\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.841500 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841482 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rx2z5" Jul 09 16:16:27.841643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.841620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81464632-b149-4ecf-b185-c77649824390-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.843580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.843556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81464632-b149-4ecf-b185-c77649824390-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.850158 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.850127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrspb\" (UniqueName: \"kubernetes.io/projected/81464632-b149-4ecf-b185-c77649824390-kube-api-access-vrspb\") pod \"insights-runtime-extractor-z8bns\" (UID: \"81464632-b149-4ecf-b185-c77649824390\") " pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.851049 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.851024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:27.939986 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.939687 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z8bns" Jul 09 16:16:27.991865 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:27.991769 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f68bbf98-jfs56"] Jul 09 16:16:27.997824 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:27.997798 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a854d5_5647_4ac3_9d40_bb1d99832b48.slice/crio-2938e1c589e3f0bcf0c9be338d262a17f3bb9aebb20a03f6969b4d401794710f WatchSource:0}: Error finding container 2938e1c589e3f0bcf0c9be338d262a17f3bb9aebb20a03f6969b4d401794710f: Status 404 returned error can't find the container with id 2938e1c589e3f0bcf0c9be338d262a17f3bb9aebb20a03f6969b4d401794710f Jul 09 16:16:28.008954 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.008915 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rx2z5"] Jul 09 16:16:28.011413 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:28.011371 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26741143_5141_49b1_96a9_7f82db5af8a7.slice/crio-b7d710a344e3735a5d38b8e4b16bbe765a9f233763431c97c5facb7eb932bb04 WatchSource:0}: Error finding container b7d710a344e3735a5d38b8e4b16bbe765a9f233763431c97c5facb7eb932bb04: Status 404 returned error can't find the container with id b7d710a344e3735a5d38b8e4b16bbe765a9f233763431c97c5facb7eb932bb04 Jul 09 16:16:28.023772 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.023695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clfl2"] Jul 09 16:16:28.027376 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:28.027352 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb89b1d_0dc3_4515_b11a_9646e3f5ebcb.slice/crio-13cfc42805ba0eb8a01128d11d92622384bd3b2c7653b46a0bf7d438aca129c2 WatchSource:0}: Error finding container 13cfc42805ba0eb8a01128d11d92622384bd3b2c7653b46a0bf7d438aca129c2: Status 404 returned error can't find the container with id 13cfc42805ba0eb8a01128d11d92622384bd3b2c7653b46a0bf7d438aca129c2 Jul 09 16:16:28.090288 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.090253 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z8bns"] Jul 09 16:16:28.093651 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:28.093617 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81464632_b149_4ecf_b185_c77649824390.slice/crio-ada7927a72b6933e4d5c77854d302fe89d08ded7ac4f36e372aae5bc65b9d6b9 WatchSource:0}: Error finding container ada7927a72b6933e4d5c77854d302fe89d08ded7ac4f36e372aae5bc65b9d6b9: Status 404 returned error can't find the container with id ada7927a72b6933e4d5c77854d302fe89d08ded7ac4f36e372aae5bc65b9d6b9 Jul 09 16:16:28.127560 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.127523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" event={"ID":"d4a854d5-5647-4ac3-9d40-bb1d99832b48","Type":"ContainerStarted","Data":"3b76246ad424f5c25587a7ce000106c03791892e72ea84391df2f7c10a03cfc8"} Jul 09 16:16:28.127739 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.127584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" event={"ID":"d4a854d5-5647-4ac3-9d40-bb1d99832b48","Type":"ContainerStarted","Data":"2938e1c589e3f0bcf0c9be338d262a17f3bb9aebb20a03f6969b4d401794710f"} Jul 09 16:16:28.127739 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.127715 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:28.128644 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.128625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z8bns" event={"ID":"81464632-b149-4ecf-b185-c77649824390","Type":"ContainerStarted","Data":"ada7927a72b6933e4d5c77854d302fe89d08ded7ac4f36e372aae5bc65b9d6b9"} Jul 09 16:16:28.129714 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.129684 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clfl2" event={"ID":"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb","Type":"ContainerStarted","Data":"13cfc42805ba0eb8a01128d11d92622384bd3b2c7653b46a0bf7d438aca129c2"} Jul 09 16:16:28.130639 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.130624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rx2z5" event={"ID":"26741143-5141-49b1-96a9-7f82db5af8a7","Type":"ContainerStarted","Data":"b7d710a344e3735a5d38b8e4b16bbe765a9f233763431c97c5facb7eb932bb04"} Jul 09 16:16:28.151471 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.151412 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" podStartSLOduration=1.151394275 podStartE2EDuration="1.151394275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:28.150391537 +0000 UTC m=+65.948612079" watchObservedRunningTime="2026-07-09 16:16:28.151394275 +0000 UTC m=+65.949614817" Jul 09 16:16:28.814625 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.814591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:16:28.814812 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.814664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:16:28.814871 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.814813 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:16:28.818092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.818066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 16:16:28.818230 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.818066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 16:16:28.819549 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.819527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k799q\"" Jul 09 16:16:28.819661 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.819584 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 16:16:28.819661 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.819594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 16:16:28.820265 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:28.820141 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c54pk\"" Jul 09 16:16:34.147978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:34.147923 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clfl2" event={"ID":"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb","Type":"ContainerStarted","Data":"572ca831806f2505f33a8039e5cdcb78c8b68b8ee62217463d03684cc70ce5e2"} Jul 09 16:16:34.149494 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:34.149463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rx2z5" event={"ID":"26741143-5141-49b1-96a9-7f82db5af8a7","Type":"ContainerStarted","Data":"ca22d315ac9cf852860a417dbc57e5023eaa5f903469f7be802c8f78b8a599eb"} Jul 09 16:16:34.166041 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:34.165776 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rx2z5" podStartSLOduration=1.161439783 podStartE2EDuration="7.165757985s" totalImagesPullingTime="6.004318202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:34.16535168 +0000 UTC m=+71.963572241" watchObservedRunningTime="2026-07-09 16:16:34.165757985 +0000 UTC m=+71.963978528" Jul 09 16:16:35.154206 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:35.154169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z8bns" event={"ID":"81464632-b149-4ecf-b185-c77649824390","Type":"ContainerStarted","Data":"3815efb54818819e0f6d4e112ba99c5d241ca89ca5645416948a903d53d68dd1"} Jul 09 16:16:35.155818 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:35.155794 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clfl2" event={"ID":"9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb","Type":"ContainerStarted","Data":"68c822e194f877cc2fb78220fce60348248cf95e439f7ca45f6c436ca35e6fd3"} Jul 09 16:16:35.175045 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:35.174993 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-clfl2" podStartSLOduration=2.185276802 podStartE2EDuration="8.174977994s" totalImagesPullingTime="5.989701192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:35.174591447 +0000 UTC m=+72.972811989" watchObservedRunningTime="2026-07-09 16:16:35.174977994 +0000 UTC m=+72.973198535" Jul 09 16:16:36.160613 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:36.160572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z8bns" event={"ID":"81464632-b149-4ecf-b185-c77649824390","Type":"ContainerStarted","Data":"6f8742d4dcafe7ea58ffcf8034cff705c95031d0d2dfbbd594b6e69f966c1bf9"} Jul 09 16:16:36.161107 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:36.160757 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:37.089418 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.089385 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8c2d" Jul 09 16:16:37.609757 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.609718 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44"] Jul 09 16:16:37.611776 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.611749 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:37.614588 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.614558 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 16:16:37.614716 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.614597 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dvbvw\"" Jul 09 16:16:37.621705 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.621679 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44"] Jul 09 16:16:37.726135 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.726093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88a078a0-4ac5-4075-aa8c-40e05e10b015-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zmz44\" (UID: \"88a078a0-4ac5-4075-aa8c-40e05e10b015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:37.826596 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:37.826558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88a078a0-4ac5-4075-aa8c-40e05e10b015-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zmz44\" (UID: \"88a078a0-4ac5-4075-aa8c-40e05e10b015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:37.826814 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:37.826729 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 16:16:37.826874 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:37.826816 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a078a0-4ac5-4075-aa8c-40e05e10b015-tls-certificates podName:88a078a0-4ac5-4075-aa8c-40e05e10b015 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:38.326793249 +0000 UTC m=+76.125013787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/88a078a0-4ac5-4075-aa8c-40e05e10b015-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-zmz44" (UID: "88a078a0-4ac5-4075-aa8c-40e05e10b015") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 16:16:38.330528 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:38.330486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88a078a0-4ac5-4075-aa8c-40e05e10b015-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zmz44\" (UID: \"88a078a0-4ac5-4075-aa8c-40e05e10b015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:38.333211 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:38.333182 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88a078a0-4ac5-4075-aa8c-40e05e10b015-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zmz44\" (UID: \"88a078a0-4ac5-4075-aa8c-40e05e10b015\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:38.524261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:38.524220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:38.849958 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:38.849927 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44"] Jul 09 16:16:38.852702 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:38.852677 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a078a0_4ac5_4075_aa8c_40e05e10b015.slice/crio-8a9bfdac2ff9c23678feb53158b255091a691d95896b6d850c692104259252b6 WatchSource:0}: Error finding container 8a9bfdac2ff9c23678feb53158b255091a691d95896b6d850c692104259252b6: Status 404 returned error can't find the container with id 8a9bfdac2ff9c23678feb53158b255091a691d95896b6d850c692104259252b6 Jul 09 16:16:39.172359 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:39.172327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z8bns" event={"ID":"81464632-b149-4ecf-b185-c77649824390","Type":"ContainerStarted","Data":"0f4e0ee97bf392ba6a1cf0d9ea7dedcb4ce931b99994c29bf5141b4daa640e5d"} Jul 09 16:16:39.173375 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:39.173354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" event={"ID":"88a078a0-4ac5-4075-aa8c-40e05e10b015","Type":"ContainerStarted","Data":"8a9bfdac2ff9c23678feb53158b255091a691d95896b6d850c692104259252b6"} Jul 09 16:16:39.191353 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:39.191307 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z8bns" podStartSLOduration=1.7010649249999998 podStartE2EDuration="12.191288564s" totalImagesPullingTime="10.490223639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:16:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:39.191051307 +0000 UTC m=+76.989271827" watchObservedRunningTime="2026-07-09 16:16:39.191288564 +0000 UTC m=+76.989509107" Jul 09 16:16:40.177982 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:40.177942 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" event={"ID":"88a078a0-4ac5-4075-aa8c-40e05e10b015","Type":"ContainerStarted","Data":"b8e90fc4c83af5fec6bc348542b9578977222fe8ff2b5e373eed69cc727a831d"} Jul 09 16:16:40.178383 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:40.178235 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:40.182747 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:40.182728 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" Jul 09 16:16:40.199601 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:40.199555 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zmz44" podStartSLOduration=1.9911896599999999 podStartE2EDuration="3.199538843s" totalImagesPullingTime="1.208349183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:40.198506221 +0000 UTC m=+77.996726765" watchObservedRunningTime="2026-07-09 16:16:40.199538843 +0000 UTC m=+77.997759390" Jul 09 16:16:46.021226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.021107 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-h954l"] Jul 09 16:16:46.023212 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.023193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.026335 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.026308 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z5pdf\"" Jul 09 16:16:46.026443 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.026330 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 16:16:46.027636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.027614 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 16:16:46.027733 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.027635 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 16:16:46.027733 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.027615 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 16:16:46.027733 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.027671 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 16:16:46.038397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.038375 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-h954l"] Jul 09 16:16:46.049980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.049955 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8v9nk"] Jul 09 16:16:46.052128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.052108 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.055348 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.055329 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 16:16:46.055348 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.055331 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 16:16:46.055603 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.055583 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 16:16:46.055685 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.055670 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6cbss\"" Jul 09 16:16:46.061499 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.061472 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rdvsm"] Jul 09 16:16:46.063326 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.063311 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.066034 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.066016 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 16:16:46.066149 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.066066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 16:16:46.066261 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.066244 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 16:16:46.066539 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.066526 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhzht\"" Jul 09 16:16:46.069961 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.069910 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8v9nk"] Jul 09 16:16:46.094955 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.094927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a08ce60f-1817-43ba-9c14-0c427fac6444-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.095151 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.094979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a08ce60f-1817-43ba-9c14-0c427fac6444-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.095151 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.094999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a08ce60f-1817-43ba-9c14-0c427fac6444-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.095151 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.095018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9rg\" (UniqueName: \"kubernetes.io/projected/a08ce60f-1817-43ba-9c14-0c427fac6444-kube-api-access-hd9rg\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.166254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.166215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-clfl2" Jul 09 16:16:46.196282 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.196551 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-sys\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.196665 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a08ce60f-1817-43ba-9c14-0c427fac6444-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.196665 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9rg\" (UniqueName: \"kubernetes.io/projected/a08ce60f-1817-43ba-9c14-0c427fac6444-kube-api-access-hd9rg\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.196665 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/753f2dd3-dccf-4251-9616-6a17b8d3b805-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.196810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sckp\" (UniqueName: \"kubernetes.io/projected/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-api-access-4sckp\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.196810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-textfile\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.196810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.196810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.196810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/753f2dd3-dccf-4251-9616-6a17b8d3b805-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196860 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-tls\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a08ce60f-1817-43ba-9c14-0c427fac6444-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-root\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.196980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5vjk\" (UniqueName: \"kubernetes.io/projected/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-kube-api-access-x5vjk\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.197021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-wtmp\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197066 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.197045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-metrics-client-ca\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197392 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.197074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a08ce60f-1817-43ba-9c14-0c427fac6444-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.197392 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.197101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.197493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.197420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a08ce60f-1817-43ba-9c14-0c427fac6444-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.200091 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.200062 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a08ce60f-1817-43ba-9c14-0c427fac6444-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.200307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.200286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a08ce60f-1817-43ba-9c14-0c427fac6444-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.205868 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.205843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9rg\" (UniqueName: \"kubernetes.io/projected/a08ce60f-1817-43ba-9c14-0c427fac6444-kube-api-access-hd9rg\") pod \"openshift-state-metrics-84bd68b896-h954l\" (UID: \"a08ce60f-1817-43ba-9c14-0c427fac6444\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.298092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/753f2dd3-dccf-4251-9616-6a17b8d3b805-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.298092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-tls\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-root\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5vjk\" (UniqueName: \"kubernetes.io/projected/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-kube-api-access-x5vjk\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-wtmp\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-metrics-client-ca\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-root\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-sys\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298372 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-wtmp\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/753f2dd3-dccf-4251-9616-6a17b8d3b805-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sckp\" (UniqueName: \"kubernetes.io/projected/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-api-access-4sckp\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-textfile\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-sys\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.298738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/753f2dd3-dccf-4251-9616-6a17b8d3b805-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.299132 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.299132 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/753f2dd3-dccf-4251-9616-6a17b8d3b805-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.299132 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.298819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-metrics-client-ca\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.299351 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.299303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-textfile\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.299520 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.299495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.299678 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.299652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.300714 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.300692 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-tls\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.300988 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.300967 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.301035 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.300982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.301035 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.301025 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.307307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.307282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sckp\" (UniqueName: \"kubernetes.io/projected/753f2dd3-dccf-4251-9616-6a17b8d3b805-kube-api-access-4sckp\") pod \"kube-state-metrics-785778ff74-8v9nk\" (UID: \"753f2dd3-dccf-4251-9616-6a17b8d3b805\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.307413 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.307340 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5vjk\" (UniqueName: \"kubernetes.io/projected/9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3-kube-api-access-x5vjk\") pod \"node-exporter-rdvsm\" (UID: \"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3\") " pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.332262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.332223 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" Jul 09 16:16:46.361216 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.361177 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" Jul 09 16:16:46.372194 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.372171 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rdvsm" Jul 09 16:16:46.384462 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:46.384412 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7c93d8_c4ea_41ad_bad5_3b4d30dadbd3.slice/crio-2807cbaee9a6ba8e922e8a4a0f5bdd17d0deadaf5ec9a29f5333760bfd03f0ef WatchSource:0}: Error finding container 2807cbaee9a6ba8e922e8a4a0f5bdd17d0deadaf5ec9a29f5333760bfd03f0ef: Status 404 returned error can't find the container with id 2807cbaee9a6ba8e922e8a4a0f5bdd17d0deadaf5ec9a29f5333760bfd03f0ef Jul 09 16:16:46.479824 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.479799 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-h954l"] Jul 09 16:16:46.482266 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:46.482240 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08ce60f_1817_43ba_9c14_0c427fac6444.slice/crio-a4bbb45f43214a4db9cdb02361899e4740c7a0bc7cd798498d765c1d3949333b WatchSource:0}: Error finding container a4bbb45f43214a4db9cdb02361899e4740c7a0bc7cd798498d765c1d3949333b: Status 404 returned error can't find the container with id a4bbb45f43214a4db9cdb02361899e4740c7a0bc7cd798498d765c1d3949333b Jul 09 16:16:46.503481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:46.503457 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-8v9nk"] Jul 09 16:16:46.511842 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:46.511814 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753f2dd3_dccf_4251_9616_6a17b8d3b805.slice/crio-27bbcbbd5c3caf244cce0ad3e65a99113dc5717bd6e2921181b4caef3c21b74c WatchSource:0}: Error finding container 27bbcbbd5c3caf244cce0ad3e65a99113dc5717bd6e2921181b4caef3c21b74c: Status 404 returned error can't find the container with id 27bbcbbd5c3caf244cce0ad3e65a99113dc5717bd6e2921181b4caef3c21b74c Jul 09 16:16:47.202565 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.202520 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" event={"ID":"753f2dd3-dccf-4251-9616-6a17b8d3b805","Type":"ContainerStarted","Data":"27bbcbbd5c3caf244cce0ad3e65a99113dc5717bd6e2921181b4caef3c21b74c"} Jul 09 16:16:47.204596 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.204557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdvsm" event={"ID":"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3","Type":"ContainerStarted","Data":"2807cbaee9a6ba8e922e8a4a0f5bdd17d0deadaf5ec9a29f5333760bfd03f0ef"} Jul 09 16:16:47.207020 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.206993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" event={"ID":"a08ce60f-1817-43ba-9c14-0c427fac6444","Type":"ContainerStarted","Data":"6918f23de718ef4064ee2770df26a1d9ad01de4396c1566f7a239a01a5ba6e91"} Jul 09 16:16:47.207167 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.207054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" event={"ID":"a08ce60f-1817-43ba-9c14-0c427fac6444","Type":"ContainerStarted","Data":"d813809722484c2eb53e82aec9c42e3ee995821943bd3d665e623e4cd015372b"} Jul 09 16:16:47.207167 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.207089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" event={"ID":"a08ce60f-1817-43ba-9c14-0c427fac6444","Type":"ContainerStarted","Data":"a4bbb45f43214a4db9cdb02361899e4740c7a0bc7cd798498d765c1d3949333b"} Jul 09 16:16:47.828190 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.828141 2562 patch_prober.go:28] interesting pod/image-registry-5f68bbf98-jfs56 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 16:16:47.828378 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:47.828214 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" podUID="d4a854d5-5647-4ac3-9d40-bb1d99832b48" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 16:16:49.137796 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.137761 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f68bbf98-jfs56" Jul 09 16:16:49.216427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.216393 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3" containerID="6922fde098fe5624a1b0c4523632053ba1f5c10351ece7e5e960e2e037015078" exitCode=0 Jul 09 16:16:49.216603 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.216479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdvsm" event={"ID":"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3","Type":"ContainerDied","Data":"6922fde098fe5624a1b0c4523632053ba1f5c10351ece7e5e960e2e037015078"} Jul 09 16:16:49.218956 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.218925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" event={"ID":"a08ce60f-1817-43ba-9c14-0c427fac6444","Type":"ContainerStarted","Data":"a34ea2cb868f0d6b134b3fad7e12b12543fcc132145c29ee2edbbffc5c8a9f42"} Jul 09 16:16:49.221314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.221289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" event={"ID":"753f2dd3-dccf-4251-9616-6a17b8d3b805","Type":"ContainerStarted","Data":"24e244fbe424ba49c2c43ebb00e4431fe291148c6ba95bbc471642a4351ce793"} Jul 09 16:16:49.221402 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.221345 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" event={"ID":"753f2dd3-dccf-4251-9616-6a17b8d3b805","Type":"ContainerStarted","Data":"04bb55caaaf3d3463a8a72e4e80c66495936877d2977ec8c30c9799af3ebb008"} Jul 09 16:16:49.221402 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.221377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" event={"ID":"753f2dd3-dccf-4251-9616-6a17b8d3b805","Type":"ContainerStarted","Data":"587d20a8906868f4027be4da5c8fd95be97f30426aa31b8c3511e5427552b396"} Jul 09 16:16:49.261767 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.261715 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-8v9nk" podStartSLOduration=1.486058113 podStartE2EDuration="3.261698477s" totalImagesPullingTime="1.775640364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:49.260453286 +0000 UTC m=+87.058673829" watchObservedRunningTime="2026-07-09 16:16:49.261698477 +0000 UTC m=+87.059919019" Jul 09 16:16:49.286058 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:49.286014 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-h954l" podStartSLOduration=2.589006539 podStartE2EDuration="4.285972029s" totalImagesPullingTime="1.69696549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:49.284633278 +0000 UTC m=+87.082853819" watchObservedRunningTime="2026-07-09 16:16:49.285972029 +0000 UTC m=+87.084192548" Jul 09 16:16:50.226683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.226641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdvsm" event={"ID":"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3","Type":"ContainerStarted","Data":"54473adf63e1f31e514dd79a8d4a8c1bf982a5037d1374f25228576595521ece"} Jul 09 16:16:50.227128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.226745 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdvsm" event={"ID":"9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3","Type":"ContainerStarted","Data":"d53f53cce6e1c90e80d96bd43531cf1e77642455ed0b5708e75f55df2cbbaed3"} Jul 09 16:16:50.250257 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.250200 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rdvsm" podStartSLOduration=2.25678296 podStartE2EDuration="4.250184736s" totalImagesPullingTime="1.901832275s" totalInitContainerRuntime="91.569501ms" isStatefulPod=true podCreationTimestamp="2026-07-09 16:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:50.249628571 +0000 UTC m=+88.047849113" watchObservedRunningTime="2026-07-09 16:16:50.250184736 +0000 UTC m=+88.048405275" Jul 09 16:16:50.824321 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.824286 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf"] Jul 09 16:16:50.826206 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.826191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:50.830197 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.830176 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 16:16:50.830327 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.830205 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hmcnm\"" Jul 09 16:16:50.847313 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.847280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf"] Jul 09 16:16:50.939723 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.939681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27299\" (UniqueName: \"kubernetes.io/projected/dc772d81-4f42-460b-93f3-286deb1a4547-kube-api-access-27299\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:50.939913 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:50.939768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc772d81-4f42-460b-93f3-286deb1a4547-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.040946 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.040903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27299\" (UniqueName: \"kubernetes.io/projected/dc772d81-4f42-460b-93f3-286deb1a4547-kube-api-access-27299\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.041124 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.040988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc772d81-4f42-460b-93f3-286deb1a4547-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.041124 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:51.041108 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 09 16:16:51.041196 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:16:51.041162 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc772d81-4f42-460b-93f3-286deb1a4547-monitoring-plugin-cert podName:dc772d81-4f42-460b-93f3-286deb1a4547 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:51.541147136 +0000 UTC m=+89.339367655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/dc772d81-4f42-460b-93f3-286deb1a4547-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-h82rf" (UID: "dc772d81-4f42-460b-93f3-286deb1a4547") : secret "monitoring-plugin-cert" not found Jul 09 16:16:51.059011 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.058973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27299\" (UniqueName: \"kubernetes.io/projected/dc772d81-4f42-460b-93f3-286deb1a4547-kube-api-access-27299\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.546014 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.545978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc772d81-4f42-460b-93f3-286deb1a4547-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.548548 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.548520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc772d81-4f42-460b-93f3-286deb1a4547-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-h82rf\" (UID: \"dc772d81-4f42-460b-93f3-286deb1a4547\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.735300 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.735253 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:51.861303 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:51.861278 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf"] Jul 09 16:16:51.863325 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:51.863288 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc772d81_4f42_460b_93f3_286deb1a4547.slice/crio-7c3e4fe391c41b87a70b0ac4291d0575479960195b9ccb450d6a2d62c426e73b WatchSource:0}: Error finding container 7c3e4fe391c41b87a70b0ac4291d0575479960195b9ccb450d6a2d62c426e73b: Status 404 returned error can't find the container with id 7c3e4fe391c41b87a70b0ac4291d0575479960195b9ccb450d6a2d62c426e73b Jul 09 16:16:52.234494 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.234459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" event={"ID":"dc772d81-4f42-460b-93f3-286deb1a4547","Type":"ContainerStarted","Data":"7c3e4fe391c41b87a70b0ac4291d0575479960195b9ccb450d6a2d62c426e73b"} Jul 09 16:16:52.276310 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.276275 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:16:52.279287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.279267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.282344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282316 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jgn9n\"" Jul 09 16:16:52.282514 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282316 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 16:16:52.282576 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282513 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 16:16:52.282616 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282587 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 16:16:52.282786 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282769 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 16:16:52.282786 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282782 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 16:16:52.282932 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282787 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 16:16:52.282932 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.282908 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 16:16:52.284356 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284338 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-edb15s7i847rf\"" Jul 09 16:16:52.284356 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284352 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 16:16:52.284532 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284429 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 16:16:52.284532 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284507 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 16:16:52.284777 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284761 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 16:16:52.284859 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284813 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 16:16:52.284859 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284849 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 16:16:52.284970 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.284900 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 16:16:52.286986 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.286958 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 16:16:52.300289 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.300258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:16:52.456510 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456510 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456735 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8j5j\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-kube-api-access-r8j5j\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456735 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-config\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456735 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456629 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456735 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-config-out\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456735 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456897 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.456980 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.456979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.457338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.457002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-web-config\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.457338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.457064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.457338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.457099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.457338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.457116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.457338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.457135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558377 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558377 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j5j\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-kube-api-access-r8j5j\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-config\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-config-out\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-web-config\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.558920 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.559699 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.559699 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.558977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.559699 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.559159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.559699 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.559211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.559699 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.559394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.559959 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.559840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.560177 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.560151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.561813 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.561478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-config\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.561813 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.561712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.561998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.561914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.563507 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.562576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.563507 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.563081 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.563507 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.563425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.563507 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.563432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.563791 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.563565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.564139 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.564115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.564399 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.564370 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.564908 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.564863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-config-out\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.565168 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.565144 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.566072 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.566051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-web-config\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.566187 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.566120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.569427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.569399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8j5j\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-kube-api-access-r8j5j\") pod \"prometheus-k8s-0\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.589745 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.589695 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:16:52.740345 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:52.740280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:16:52.742798 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:16:52.742768 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5756e486_c7af_4d7c_bafc_c34bc2948039.slice/crio-4897a962d711901ebf58d38c63959082f103a1d029e249ef68e22f10ecbcfa94 WatchSource:0}: Error finding container 4897a962d711901ebf58d38c63959082f103a1d029e249ef68e22f10ecbcfa94: Status 404 returned error can't find the container with id 4897a962d711901ebf58d38c63959082f103a1d029e249ef68e22f10ecbcfa94 Jul 09 16:16:53.238985 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:53.238951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"4897a962d711901ebf58d38c63959082f103a1d029e249ef68e22f10ecbcfa94"} Jul 09 16:16:54.243608 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:54.243571 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92" exitCode=0 Jul 09 16:16:54.244191 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:54.243662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92"} Jul 09 16:16:54.245332 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:54.245303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" event={"ID":"dc772d81-4f42-460b-93f3-286deb1a4547","Type":"ContainerStarted","Data":"939f0d05f66c607d2fd5efa001a4f438b946d4d59ecbc58fa2e6a23801438a29"} Jul 09 16:16:54.245536 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:54.245511 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:54.250303 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:54.250272 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" Jul 09 16:16:54.300086 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:54.300030 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-h82rf" podStartSLOduration=2.8563113749999998 podStartE2EDuration="4.300013043s" totalImagesPullingTime="1.443701668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:54.299553771 +0000 UTC m=+92.097774314" watchObservedRunningTime="2026-07-09 16:16:54.300013043 +0000 UTC m=+92.098233584" Jul 09 16:16:59.269179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:59.269144 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e"} Jul 09 16:16:59.269569 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:16:59.269218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d"} Jul 09 16:17:01.281532 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:01.281503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88"} Jul 09 16:17:01.281932 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:01.281583 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90"} Jul 09 16:17:02.288180 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:02.288138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81"} Jul 09 16:17:02.288558 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:02.288255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerStarted","Data":"bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf"} Jul 09 16:17:02.320549 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:02.320496 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.420620124 podStartE2EDuration="10.320478212s" totalImagesPullingTime="7.880507245s" totalInitContainerRuntime="19.350843ms" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:02.318539981 +0000 UTC m=+100.116760541" watchObservedRunningTime="2026-07-09 16:17:02.320478212 +0000 UTC m=+100.118698781" Jul 09 16:17:02.590552 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:02.590463 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:17:31.611530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.611489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:17:31.612101 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.611550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:17:31.612101 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.611579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:17:31.615013 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.614993 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 16:17:31.616084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.616070 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 16:17:31.616141 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.616072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 16:17:31.624092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.624062 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d7c05a8-7911-461d-a206-57df68595e5e-metrics-certs\") pod \"network-metrics-daemon-mpbbn\" (UID: \"8d7c05a8-7911-461d-a206-57df68595e5e\") " pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:17:31.624193 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.624162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1451f3-2475-4614-b6a8-df41425e93b8-original-pull-secret\") pod \"global-pull-secret-syncer-b8dkn\" (UID: \"8b1451f3-2475-4614-b6a8-df41425e93b8\") " pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:17:31.624445 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.624425 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 16:17:31.634068 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.634038 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27sds\" (UniqueName: \"kubernetes.io/projected/174ae44f-0ade-475f-b740-c6606d184c02-kube-api-access-27sds\") pod \"network-check-target-xlhnm\" (UID: \"174ae44f-0ade-475f-b740-c6606d184c02\") " pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:17:31.831431 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.831206 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c54pk\"" Jul 09 16:17:31.837786 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.837764 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k799q\"" Jul 09 16:17:31.838787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.838763 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:17:31.841419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.841397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8dkn" Jul 09 16:17:31.845358 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:31.845324 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpbbn" Jul 09 16:17:32.029684 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:32.028472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xlhnm"] Jul 09 16:17:32.035167 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:17:32.035131 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174ae44f_0ade_475f_b740_c6606d184c02.slice/crio-6e267f9f80716edcbfc2e24353bfb59c9534a5635453a5faef3c6557adb190d2 WatchSource:0}: Error finding container 6e267f9f80716edcbfc2e24353bfb59c9534a5635453a5faef3c6557adb190d2: Status 404 returned error can't find the container with id 6e267f9f80716edcbfc2e24353bfb59c9534a5635453a5faef3c6557adb190d2 Jul 09 16:17:32.264014 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:32.263984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8dkn"] Jul 09 16:17:32.266456 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:17:32.266418 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1451f3_2475_4614_b6a8_df41425e93b8.slice/crio-d810158727eed0369f57dc66a7d0930bd8d09766a276c4d782c7aef9646155aa WatchSource:0}: Error finding container d810158727eed0369f57dc66a7d0930bd8d09766a276c4d782c7aef9646155aa: Status 404 returned error can't find the container with id d810158727eed0369f57dc66a7d0930bd8d09766a276c4d782c7aef9646155aa Jul 09 16:17:32.267270 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:32.267055 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpbbn"] Jul 09 16:17:32.269143 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:17:32.269115 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7c05a8_7911_461d_a206_57df68595e5e.slice/crio-bf1e2ecf3649a18bede8d9c764bf7016cfd5cd3c0ac649113a01143c3501fb4b WatchSource:0}: Error finding container bf1e2ecf3649a18bede8d9c764bf7016cfd5cd3c0ac649113a01143c3501fb4b: Status 404 returned error can't find the container with id bf1e2ecf3649a18bede8d9c764bf7016cfd5cd3c0ac649113a01143c3501fb4b Jul 09 16:17:32.398668 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:32.398614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8dkn" event={"ID":"8b1451f3-2475-4614-b6a8-df41425e93b8","Type":"ContainerStarted","Data":"d810158727eed0369f57dc66a7d0930bd8d09766a276c4d782c7aef9646155aa"} Jul 09 16:17:32.400262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:32.400225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpbbn" event={"ID":"8d7c05a8-7911-461d-a206-57df68595e5e","Type":"ContainerStarted","Data":"bf1e2ecf3649a18bede8d9c764bf7016cfd5cd3c0ac649113a01143c3501fb4b"} Jul 09 16:17:32.401694 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:32.401662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xlhnm" event={"ID":"174ae44f-0ade-475f-b740-c6606d184c02","Type":"ContainerStarted","Data":"6e267f9f80716edcbfc2e24353bfb59c9534a5635453a5faef3c6557adb190d2"} Jul 09 16:17:34.413492 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:34.413449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpbbn" event={"ID":"8d7c05a8-7911-461d-a206-57df68595e5e","Type":"ContainerStarted","Data":"9e837644f15e215dbec3f99abc03a7fed1db5cc5de0d2c57762c107559d87aa4"} Jul 09 16:17:34.413896 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:34.413517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpbbn" event={"ID":"8d7c05a8-7911-461d-a206-57df68595e5e","Type":"ContainerStarted","Data":"5ac0202e023901470ea171e2d557eebac6d1793eb456e1c705483155fc098642"} Jul 09 16:17:34.432101 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:34.432044 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mpbbn" podStartSLOduration=130.009632016 podStartE2EDuration="2m11.432020307s" totalImagesPullingTime="1.422388291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:34.430750779 +0000 UTC m=+132.228971339" watchObservedRunningTime="2026-07-09 16:17:34.432020307 +0000 UTC m=+132.230240850" Jul 09 16:17:38.429243 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:38.429192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8dkn" event={"ID":"8b1451f3-2475-4614-b6a8-df41425e93b8","Type":"ContainerStarted","Data":"e9445b5d55fbefd5e0944644fcdee58faa264a894a3f3232bcc468f89683160b"} Jul 09 16:17:38.430695 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:38.430669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xlhnm" event={"ID":"174ae44f-0ade-475f-b740-c6606d184c02","Type":"ContainerStarted","Data":"50020ed16571a2df7a2837f455ab4411a6adbf5432f3e7a7f8f562bb52bdffba"} Jul 09 16:17:38.430811 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:38.430720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:17:38.445990 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:38.445941 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b8dkn" podStartSLOduration=130.142356562 podStartE2EDuration="2m15.445928196s" totalImagesPullingTime="5.303571634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:38.445340258 +0000 UTC m=+136.243560801" watchObservedRunningTime="2026-07-09 16:17:38.445928196 +0000 UTC m=+136.244148737" Jul 09 16:17:38.460848 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:38.460774 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xlhnm" podStartSLOduration=129.928080098 podStartE2EDuration="2m15.46075661s" totalImagesPullingTime="5.532676512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:38.460230716 +0000 UTC m=+136.258451260" watchObservedRunningTime="2026-07-09 16:17:38.46075661 +0000 UTC m=+136.258977154" Jul 09 16:17:52.590245 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:52.590207 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:17:52.610778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:52.610751 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:17:53.501115 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:17:53.501087 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:09.437281 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:09.437251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xlhnm" Jul 09 16:18:10.641434 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641392 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:18:10.641925 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641793 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="prometheus" containerID="cri-o://02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" gracePeriod=600 Jul 09 16:18:10.641925 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641803 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-web" containerID="cri-o://87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88" gracePeriod=600 Jul 09 16:18:10.641925 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641849 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy" containerID="cri-o://bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf" gracePeriod=600 Jul 09 16:18:10.641925 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641810 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="config-reloader" containerID="cri-o://159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e" gracePeriod=600 Jul 09 16:18:10.642149 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641927 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="thanos-sidecar" containerID="cri-o://ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90" gracePeriod=600 Jul 09 16:18:10.642149 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:10.641812 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81" gracePeriod=600 Jul 09 16:18:11.556646 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:11.556610 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88" exitCode=0 Jul 09 16:18:11.556646 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:11.556636 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90" exitCode=0 Jul 09 16:18:11.556646 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:11.556644 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e" exitCode=0 Jul 09 16:18:11.556872 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:11.556681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88"} Jul 09 16:18:11.556872 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:11.556744 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90"} Jul 09 16:18:11.556872 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:11.556773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e"} Jul 09 16:18:12.563924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:12.563872 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81" exitCode=0 Jul 09 16:18:12.563924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:12.563914 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf" exitCode=0 Jul 09 16:18:12.563924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:12.563911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81"} Jul 09 16:18:12.564449 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:12.563981 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf"} Jul 09 16:18:13.487609 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:13.487546 2562 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 09 16:18:13.488621 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:13.488590 2562 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 09 16:18:13.489478 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:13.489448 2562 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 09 16:18:13.489576 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:13.489491 2562 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="prometheus" probeResult="unknown" Jul 09 16:18:18.486996 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.486938 2562 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d is running failed: container process not found" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 09 16:18:18.487469 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.487289 2562 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d is running failed: container process not found" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 09 16:18:18.487670 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.487643 2562 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d is running failed: container process not found" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 09 16:18:18.487729 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.487682 2562 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d is running failed: container process not found" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="prometheus" probeResult="unknown" Jul 09 16:18:18.584037 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.584010 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:18.589418 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.589387 2562 generic.go:356] "Generic (PLEG): container finished" podID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" exitCode=0 Jul 09 16:18:18.589574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.589474 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:18.589574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.589469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d"} Jul 09 16:18:18.589574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.589558 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5756e486-c7af-4d7c-bafc-c34bc2948039","Type":"ContainerDied","Data":"4897a962d711901ebf58d38c63959082f103a1d029e249ef68e22f10ecbcfa94"} Jul 09 16:18:18.589698 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.589609 2562 scope.go:117] "RemoveContainer" containerID="2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81" Jul 09 16:18:18.597104 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.597082 2562 scope.go:117] "RemoveContainer" containerID="bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf" Jul 09 16:18:18.603986 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.603960 2562 scope.go:117] "RemoveContainer" containerID="87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88" Jul 09 16:18:18.610284 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.610266 2562 scope.go:117] "RemoveContainer" containerID="ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90" Jul 09 16:18:18.618312 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.618292 2562 scope.go:117] "RemoveContainer" containerID="159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e" Jul 09 16:18:18.625764 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.625740 2562 scope.go:117] "RemoveContainer" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" Jul 09 16:18:18.633216 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.633199 2562 scope.go:117] "RemoveContainer" containerID="d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92" Jul 09 16:18:18.639912 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.639851 2562 scope.go:117] "RemoveContainer" containerID="2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81" Jul 09 16:18:18.640159 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.640141 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81\": container with ID starting with 2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81 not found: ID does not exist" containerID="2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81" Jul 09 16:18:18.640237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.640168 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81"} err="failed to get container status \"2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81\": rpc error: code = NotFound desc = could not find container \"2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81\": container with ID starting with 2bd584081025e22707e66da279be56d029a272a02f36a9fdd3fb2f054aae5f81 not found: ID does not exist" Jul 09 16:18:18.640237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.640202 2562 scope.go:117] "RemoveContainer" containerID="bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf" Jul 09 16:18:18.640453 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.640436 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf\": container with ID starting with bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf not found: ID does not exist" containerID="bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf" Jul 09 16:18:18.640508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.640459 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf"} err="failed to get container status \"bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf\": rpc error: code = NotFound desc = could not find container \"bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf\": container with ID starting with bf58b239671a30cefd32248fbce34957c2f2b31de08b06dca62e1e4ee54c66bf not found: ID does not exist" Jul 09 16:18:18.640508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.640473 2562 scope.go:117] "RemoveContainer" containerID="87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88" Jul 09 16:18:18.640746 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.640729 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88\": container with ID starting with 87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88 not found: ID does not exist" containerID="87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88" Jul 09 16:18:18.640808 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.640752 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88"} err="failed to get container status \"87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88\": rpc error: code = NotFound desc = could not find container \"87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88\": container with ID starting with 87d8fe579172abc66cfc779d170c13e8ba5f4832a51cfbd159f19f684327fd88 not found: ID does not exist" Jul 09 16:18:18.640808 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.640769 2562 scope.go:117] "RemoveContainer" containerID="ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90" Jul 09 16:18:18.641044 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.641029 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90\": container with ID starting with ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90 not found: ID does not exist" containerID="ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90" Jul 09 16:18:18.641098 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641048 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90"} err="failed to get container status \"ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90\": rpc error: code = NotFound desc = could not find container \"ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90\": container with ID starting with ae51584c85f1585b0082a688fb046fbbd34a02d9de7f6f048a1b3c101520ac90 not found: ID does not exist" Jul 09 16:18:18.641098 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641066 2562 scope.go:117] "RemoveContainer" containerID="159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e" Jul 09 16:18:18.641312 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.641292 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e\": container with ID starting with 159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e not found: ID does not exist" containerID="159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e" Jul 09 16:18:18.641365 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641316 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e"} err="failed to get container status \"159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e\": rpc error: code = NotFound desc = could not find container \"159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e\": container with ID starting with 159676ec5c47a06d590666f3bd7d17fc1c23b790c66043d7f2f0699e26d85a6e not found: ID does not exist" Jul 09 16:18:18.641365 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641330 2562 scope.go:117] "RemoveContainer" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" Jul 09 16:18:18.641538 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.641523 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d\": container with ID starting with 02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d not found: ID does not exist" containerID="02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d" Jul 09 16:18:18.641593 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641542 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d"} err="failed to get container status \"02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d\": rpc error: code = NotFound desc = could not find container \"02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d\": container with ID starting with 02066448184bb70bc1a07c9821400d6d5da1f12637e55321fe9ce8d8d56d573d not found: ID does not exist" Jul 09 16:18:18.641593 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641553 2562 scope.go:117] "RemoveContainer" containerID="d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92" Jul 09 16:18:18.641782 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:18:18.641764 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92\": container with ID starting with d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92 not found: ID does not exist" containerID="d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92" Jul 09 16:18:18.641852 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.641787 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92"} err="failed to get container status \"d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92\": rpc error: code = NotFound desc = could not find container \"d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92\": container with ID starting with d14665bfff34fee81a3ecb112951415f87e99085a23c296f2843599e23e34a92 not found: ID does not exist" Jul 09 16:18:18.711758 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.711725 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.711758 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.711759 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-1\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712026 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.711784 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-metrics-client-certs\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712026 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.711819 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-thanos-prometheus-http-client-file\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712026 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.711981 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-kubelet-serving-ca-bundle\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712035 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-kube-rbac-proxy\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712066 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8j5j\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-kube-api-access-r8j5j\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712115 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-0\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712139 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-trusted-ca-bundle\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712146 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.712179 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712178 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-serving-certs-ca-bundle\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712213 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-2\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.712476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712258 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-db\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712564 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712618 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-grpc-tls\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712655 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-config-out\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-tls\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-tls-assets\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712854 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712905 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-config\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712941 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-metrics-client-ca\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712947 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-web-config\") pod \"5756e486-c7af-4d7c-bafc-c34bc2948039\" (UID: \"5756e486-c7af-4d7c-bafc-c34bc2948039\") " Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.713213 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.713235 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.713251 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.712957 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.713204 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.713336 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.713998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.713730 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:18:18.715672 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.715598 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:18:18.716429 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.716338 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.716793 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.716757 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.716793 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.716774 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-kube-api-access-r8j5j" (OuterVolumeSpecName: "kube-api-access-r8j5j") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "kube-api-access-r8j5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:18:18.716967 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.716931 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.717215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717036 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.717215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717125 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.717215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717162 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.717356 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717223 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-config-out" (OuterVolumeSpecName: "config-out") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:18:18.717456 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717428 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-config" (OuterVolumeSpecName: "config") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.717783 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717763 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.718017 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.717999 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:18:18.727208 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.727188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-web-config" (OuterVolumeSpecName: "web-config") pod "5756e486-c7af-4d7c-bafc-c34bc2948039" (UID: "5756e486-c7af-4d7c-bafc-c34bc2948039"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:18:18.814371 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814335 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814371 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814363 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-metrics-client-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814371 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814373 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814383 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-kube-rbac-proxy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814392 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8j5j\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-kube-api-access-r8j5j\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814402 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814411 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814419 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814427 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-prometheus-k8s-db\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814435 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-grpc-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814443 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5756e486-c7af-4d7c-bafc-c34bc2948039-config-out\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814450 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814460 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5756e486-c7af-4d7c-bafc-c34bc2948039-tls-assets\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814468 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814477 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-config\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814485 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5756e486-c7af-4d7c-bafc-c34bc2948039-configmap-metrics-client-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.814598 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.814494 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5756e486-c7af-4d7c-bafc-c34bc2948039-web-config\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:18:18.913440 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.913407 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:18:18.917801 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.917777 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:18:18.944025 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.943996 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:18:18.944313 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944296 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="thanos-sidecar" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944316 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="thanos-sidecar" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944330 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="config-reloader" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944338 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="config-reloader" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944351 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="prometheus" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944360 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="prometheus" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944371 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="init-config-reloader" Jul 09 16:18:18.944385 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944380 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="init-config-reloader" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944395 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-thanos" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944404 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-thanos" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944421 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-web" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944430 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-web" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944440 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944447 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944506 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944518 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="prometheus" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944527 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="config-reloader" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944536 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="thanos-sidecar" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944544 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-thanos" Jul 09 16:18:18.944704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.944553 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" containerName="kube-rbac-proxy-web" Jul 09 16:18:18.948430 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.948409 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:18.951337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.951320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 16:18:18.951530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.951515 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jgn9n\"" Jul 09 16:18:18.951581 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.951544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 16:18:18.951629 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.951610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 16:18:18.951671 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.951644 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 16:18:18.951739 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.951727 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 16:18:18.952085 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952070 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 16:18:18.952085 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952075 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 16:18:18.952204 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952082 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 16:18:18.952451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952432 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 16:18:18.952537 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952470 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 16:18:18.952537 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952438 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 16:18:18.952642 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-edb15s7i847rf\"" Jul 09 16:18:18.952989 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.952975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 16:18:18.953060 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.953045 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 16:18:18.955905 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.955870 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 16:18:18.958178 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.958162 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 16:18:18.966352 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:18.966295 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:18:19.117299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a2bf0ec-2631-4108-92aa-082125631266-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-config\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117480 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a2bf0ec-2631-4108-92aa-082125631266-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117604 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5l6r\" (UniqueName: \"kubernetes.io/projected/5a2bf0ec-2631-4108-92aa-082125631266-kube-api-access-g5l6r\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117682 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.117787 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.117780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.218866 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.218777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.218866 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.218815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.218866 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.218838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a2bf0ec-2631-4108-92aa-082125631266-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.218866 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.218854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.218999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5l6r\" (UniqueName: \"kubernetes.io/projected/5a2bf0ec-2631-4108-92aa-082125631266-kube-api-access-g5l6r\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a2bf0ec-2631-4108-92aa-082125631266-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219425 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-config\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.219573 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219571 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.220119 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.220119 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.219796 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.220308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.220283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.220595 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.220569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.221119 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.221094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.221747 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.221725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a2bf0ec-2631-4108-92aa-082125631266-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.222201 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.222176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.222201 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.222182 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.222340 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.222240 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a2bf0ec-2631-4108-92aa-082125631266-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.222399 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.222355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.223195 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.223169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.223386 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.223361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.223549 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.223531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.224359 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.224330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-config\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.224756 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.224737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.224806 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.224794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a2bf0ec-2631-4108-92aa-082125631266-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.225065 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.225048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a2bf0ec-2631-4108-92aa-082125631266-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.228891 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.228858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5l6r\" (UniqueName: \"kubernetes.io/projected/5a2bf0ec-2631-4108-92aa-082125631266-kube-api-access-g5l6r\") pod \"prometheus-k8s-0\" (UID: \"5a2bf0ec-2631-4108-92aa-082125631266\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.258849 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.258823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:18:19.405057 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.405025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 16:18:19.409593 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:18:19.409556 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2bf0ec_2631_4108_92aa_082125631266.slice/crio-5704fe374a31687d41f8e0cabc45dd08c2943176b315c14909205eada054d903 WatchSource:0}: Error finding container 5704fe374a31687d41f8e0cabc45dd08c2943176b315c14909205eada054d903: Status 404 returned error can't find the container with id 5704fe374a31687d41f8e0cabc45dd08c2943176b315c14909205eada054d903 Jul 09 16:18:19.594828 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.594790 2562 generic.go:356] "Generic (PLEG): container finished" podID="5a2bf0ec-2631-4108-92aa-082125631266" containerID="a0aff08f4a13215a66f6afd7d422ce77bd75dcdf71ebb1a1e144b659fa7b5c8d" exitCode=0 Jul 09 16:18:19.595220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.594863 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerDied","Data":"a0aff08f4a13215a66f6afd7d422ce77bd75dcdf71ebb1a1e144b659fa7b5c8d"} Jul 09 16:18:19.595220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:19.594944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"5704fe374a31687d41f8e0cabc45dd08c2943176b315c14909205eada054d903"} Jul 09 16:18:20.602132 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.602093 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"6c672c60fb4edd2219ae858db2a9ff062611694ce1a41e023accf560e28220be"} Jul 09 16:18:20.602502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.602186 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"cec629095ececdef671081f3b1a2c6fdb9b2b1f4f0b9f68e5128cfed5b1f800d"} Jul 09 16:18:20.602502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.602213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"974d60d63f11f8402ca3ef0d045aedcb68a6ef18a2757fbfd0f6cadd12e6e649"} Jul 09 16:18:20.602502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.602234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"04997630809c7d7b038d0a61c355617a5ffcb8b80a708218e5d218a23e2829ad"} Jul 09 16:18:20.602502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.602257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"7900a7a67587989254e69ee89d71bc6643deeb5c155dd123cda328a41f05b9bf"} Jul 09 16:18:20.602502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.602279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a2bf0ec-2631-4108-92aa-082125631266","Type":"ContainerStarted","Data":"c6ae4208ee1e597ca34801d05a89f45c1077094879640fc022cf1ff96e290295"} Jul 09 16:18:20.639000 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.638936 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.620002056 podStartE2EDuration="2.638917638s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.915582ms" isStatefulPod=false podCreationTimestamp="2026-07-09 16:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:18:20.636571887 +0000 UTC m=+178.434792439" watchObservedRunningTime="2026-07-09 16:18:20.638917638 +0000 UTC m=+178.437138179" Jul 09 16:18:20.821297 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:20.821264 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5756e486-c7af-4d7c-bafc-c34bc2948039" path="/var/lib/kubelet/pods/5756e486-c7af-4d7c-bafc-c34bc2948039/volumes" Jul 09 16:18:24.259738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:18:24.259705 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:19:19.259937 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:19:19.259902 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:19:19.274455 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:19:19.274423 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:19:19.827454 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:19:19.827428 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 16:20:22.655461 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:22.655426 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:20:22.656022 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:22.655561 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:20:22.663223 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:22.663202 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 16:20:49.260276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.260237 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp"] Jul 09 16:20:49.262228 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.262212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.266190 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.266165 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 16:20:49.267572 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.267553 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-w8bfs\"" Jul 09 16:20:49.267688 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.267564 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:20:49.273684 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.273661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp"] Jul 09 16:20:49.368774 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.368740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwp9c\" (UniqueName: \"kubernetes.io/projected/392bca38-51fa-4cbc-9db0-b6dbf77712d7-kube-api-access-bwp9c\") pod \"openshift-lws-operator-fd8ccff4c-5x5rp\" (UID: \"392bca38-51fa-4cbc-9db0-b6dbf77712d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.368974 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.368783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392bca38-51fa-4cbc-9db0-b6dbf77712d7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5x5rp\" (UID: \"392bca38-51fa-4cbc-9db0-b6dbf77712d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.470048 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.470011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwp9c\" (UniqueName: \"kubernetes.io/projected/392bca38-51fa-4cbc-9db0-b6dbf77712d7-kube-api-access-bwp9c\") pod \"openshift-lws-operator-fd8ccff4c-5x5rp\" (UID: \"392bca38-51fa-4cbc-9db0-b6dbf77712d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.470260 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.470060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392bca38-51fa-4cbc-9db0-b6dbf77712d7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5x5rp\" (UID: \"392bca38-51fa-4cbc-9db0-b6dbf77712d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.470390 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.470372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392bca38-51fa-4cbc-9db0-b6dbf77712d7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5x5rp\" (UID: \"392bca38-51fa-4cbc-9db0-b6dbf77712d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.479637 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.479604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwp9c\" (UniqueName: \"kubernetes.io/projected/392bca38-51fa-4cbc-9db0-b6dbf77712d7-kube-api-access-bwp9c\") pod \"openshift-lws-operator-fd8ccff4c-5x5rp\" (UID: \"392bca38-51fa-4cbc-9db0-b6dbf77712d7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.577817 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.577715 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" Jul 09 16:20:49.708687 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.708661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp"] Jul 09 16:20:49.711051 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:20:49.711026 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod392bca38_51fa_4cbc_9db0_b6dbf77712d7.slice/crio-55c5ce67c08e2e0ce376dd7e232102b472a44738a6f1686688bb2300a81e81bb WatchSource:0}: Error finding container 55c5ce67c08e2e0ce376dd7e232102b472a44738a6f1686688bb2300a81e81bb: Status 404 returned error can't find the container with id 55c5ce67c08e2e0ce376dd7e232102b472a44738a6f1686688bb2300a81e81bb Jul 09 16:20:49.712441 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:49.712426 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:20:50.120519 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:50.120487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" event={"ID":"392bca38-51fa-4cbc-9db0-b6dbf77712d7","Type":"ContainerStarted","Data":"55c5ce67c08e2e0ce376dd7e232102b472a44738a6f1686688bb2300a81e81bb"} Jul 09 16:20:53.133817 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:53.133780 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" event={"ID":"392bca38-51fa-4cbc-9db0-b6dbf77712d7","Type":"ContainerStarted","Data":"522d3f58da2046722be6ff3d9188b67cbedec751a5d30333fa26dcdb1a64bc8b"} Jul 09 16:20:53.159947 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:20:53.159868 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5x5rp" podStartSLOduration=1.6331467160000002 podStartE2EDuration="4.159849916s" totalImagesPullingTime="2.5267032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:53.158525002 +0000 UTC m=+330.956745543" watchObservedRunningTime="2026-07-09 16:20:53.159849916 +0000 UTC m=+330.958070456" Jul 09 16:21:13.340652 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.340621 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr"] Jul 09 16:21:13.343073 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.343052 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.346032 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.346012 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 16:21:13.346410 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.346370 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 16:21:13.346510 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.346492 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gdnnq\"" Jul 09 16:21:13.346575 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.346499 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 16:21:13.347523 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.347510 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 16:21:13.366988 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.366966 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr"] Jul 09 16:21:13.460685 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.460657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/016b5be6-0fcf-4325-ab16-c0063ba67258-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.460876 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.460692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/016b5be6-0fcf-4325-ab16-c0063ba67258-manifests\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.460876 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.460720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016b5be6-0fcf-4325-ab16-c0063ba67258-tmp\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.460876 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.460840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrqs\" (UniqueName: \"kubernetes.io/projected/016b5be6-0fcf-4325-ab16-c0063ba67258-kube-api-access-sdrqs\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.461061 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.460934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/016b5be6-0fcf-4325-ab16-c0063ba67258-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.561952 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.561917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/016b5be6-0fcf-4325-ab16-c0063ba67258-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.562137 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.561974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/016b5be6-0fcf-4325-ab16-c0063ba67258-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.562137 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.562097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/016b5be6-0fcf-4325-ab16-c0063ba67258-manifests\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.562250 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.562142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016b5be6-0fcf-4325-ab16-c0063ba67258-tmp\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.562250 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.562197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrqs\" (UniqueName: \"kubernetes.io/projected/016b5be6-0fcf-4325-ab16-c0063ba67258-kube-api-access-sdrqs\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.562537 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.562504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/016b5be6-0fcf-4325-ab16-c0063ba67258-manifests\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.564324 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.564299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016b5be6-0fcf-4325-ab16-c0063ba67258-tmp\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.564447 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.564405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/016b5be6-0fcf-4325-ab16-c0063ba67258-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.564507 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.564490 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/016b5be6-0fcf-4325-ab16-c0063ba67258-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.574395 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.574374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrqs\" (UniqueName: \"kubernetes.io/projected/016b5be6-0fcf-4325-ab16-c0063ba67258-kube-api-access-sdrqs\") pod \"opendatahub-operator-controller-manager-7df8c55954-tr5kr\" (UID: \"016b5be6-0fcf-4325-ab16-c0063ba67258\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.654402 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.654376 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:13.802776 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:13.802752 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr"] Jul 09 16:21:13.804920 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:21:13.804890 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016b5be6_0fcf_4325_ab16_c0063ba67258.slice/crio-97706b1c6c21309a7fe2f4e90eeec77d60b9c064bf3bfd2baab94fb7ea833938 WatchSource:0}: Error finding container 97706b1c6c21309a7fe2f4e90eeec77d60b9c064bf3bfd2baab94fb7ea833938: Status 404 returned error can't find the container with id 97706b1c6c21309a7fe2f4e90eeec77d60b9c064bf3bfd2baab94fb7ea833938 Jul 09 16:21:14.206929 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:14.206870 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" event={"ID":"016b5be6-0fcf-4325-ab16-c0063ba67258","Type":"ContainerStarted","Data":"97706b1c6c21309a7fe2f4e90eeec77d60b9c064bf3bfd2baab94fb7ea833938"} Jul 09 16:21:17.221013 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:17.220976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" event={"ID":"016b5be6-0fcf-4325-ab16-c0063ba67258","Type":"ContainerStarted","Data":"10afef05ef9c0958f3c535cfffe787e9fbff5b959528c4fb26e13cebada9786c"} Jul 09 16:21:19.231206 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:19.231171 2562 generic.go:356] "Generic (PLEG): container finished" podID="016b5be6-0fcf-4325-ab16-c0063ba67258" containerID="10afef05ef9c0958f3c535cfffe787e9fbff5b959528c4fb26e13cebada9786c" exitCode=0 Jul 09 16:21:19.231607 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:19.231219 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" event={"ID":"016b5be6-0fcf-4325-ab16-c0063ba67258","Type":"ContainerDied","Data":"10afef05ef9c0958f3c535cfffe787e9fbff5b959528c4fb26e13cebada9786c"} Jul 09 16:21:20.236582 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:20.236547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" event={"ID":"016b5be6-0fcf-4325-ab16-c0063ba67258","Type":"ContainerStarted","Data":"947cf5afeeb4156516767747a3365548f1aeb7b93058339fd1497f70dcb1c12d"} Jul 09 16:21:20.237009 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:20.236776 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:20.265677 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:20.265625 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" podStartSLOduration=2.342682963 podStartE2EDuration="7.265607237s" totalImagesPullingTime="3.748826133s" totalInitContainerRuntime="1.174098141s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:20.265031113 +0000 UTC m=+358.063251666" watchObservedRunningTime="2026-07-09 16:21:20.265607237 +0000 UTC m=+358.063827778" Jul 09 16:21:31.242227 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:31.242192 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-tr5kr" Jul 09 16:21:33.975343 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.975312 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl"] Jul 09 16:21:33.989213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.989182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:33.992340 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.992320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 16:21:33.992340 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.992327 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 16:21:33.992539 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.992347 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 16:21:33.992577 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.992554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 16:21:33.993770 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:33.993752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-67j44\"" Jul 09 16:21:34.001085 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.001057 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl"] Jul 09 16:21:34.140635 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.140594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tls-certs\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.140635 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.140631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45gw\" (UniqueName: \"kubernetes.io/projected/6c653629-ab5b-4dd5-a29c-5384f723cb7d-kube-api-access-t45gw\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.140850 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.140662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tmp\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.241582 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.241492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tmp\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.241582 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.241567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tls-certs\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.241964 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.241593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45gw\" (UniqueName: \"kubernetes.io/projected/6c653629-ab5b-4dd5-a29c-5384f723cb7d-kube-api-access-t45gw\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.241964 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:21:34.241705 2562 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 09 16:21:34.241964 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:21:34.241786 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tls-certs podName:6c653629-ab5b-4dd5-a29c-5384f723cb7d nodeName:}" failed. No retries permitted until 2026-07-09 16:21:34.741765764 +0000 UTC m=+372.539986293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tls-certs") pod "kube-auth-proxy-59d9878b65-pcrrl" (UID: "6c653629-ab5b-4dd5-a29c-5384f723cb7d") : secret "kube-auth-proxy-tls" not found Jul 09 16:21:34.243952 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.243931 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tmp\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.251943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.251917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45gw\" (UniqueName: \"kubernetes.io/projected/6c653629-ab5b-4dd5-a29c-5384f723cb7d-kube-api-access-t45gw\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.745240 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.745204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tls-certs\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.750271 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.750231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c653629-ab5b-4dd5-a29c-5384f723cb7d-tls-certs\") pod \"kube-auth-proxy-59d9878b65-pcrrl\" (UID: \"6c653629-ab5b-4dd5-a29c-5384f723cb7d\") " pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:34.898930 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:34.898861 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" Jul 09 16:21:35.033113 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:35.033090 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl"] Jul 09 16:21:35.035399 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:21:35.035371 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c653629_ab5b_4dd5_a29c_5384f723cb7d.slice/crio-6b985fed0dd8f3627d25c4850898cbb56c7fdb4467736f9517f657e70efbf804 WatchSource:0}: Error finding container 6b985fed0dd8f3627d25c4850898cbb56c7fdb4467736f9517f657e70efbf804: Status 404 returned error can't find the container with id 6b985fed0dd8f3627d25c4850898cbb56c7fdb4467736f9517f657e70efbf804 Jul 09 16:21:35.292213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:35.292123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" event={"ID":"6c653629-ab5b-4dd5-a29c-5384f723cb7d","Type":"ContainerStarted","Data":"6b985fed0dd8f3627d25c4850898cbb56c7fdb4467736f9517f657e70efbf804"} Jul 09 16:21:38.305109 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:38.305072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" event={"ID":"6c653629-ab5b-4dd5-a29c-5384f723cb7d","Type":"ContainerStarted","Data":"aaae9db867d4931ecff159fb1b909a3cfa843c8a0ee2d282380d91d28acc017c"} Jul 09 16:21:38.326345 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:38.326289 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59d9878b65-pcrrl" podStartSLOduration=2.265003559 podStartE2EDuration="5.326271135s" totalImagesPullingTime="3.061267576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:38.325927016 +0000 UTC m=+376.124147578" watchObservedRunningTime="2026-07-09 16:21:38.326271135 +0000 UTC m=+376.124491678" Jul 09 16:21:41.783319 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.783285 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5"] Jul 09 16:21:41.786601 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.786581 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.790349 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.790325 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xrnh9\"" Jul 09 16:21:41.790448 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.790346 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 16:21:41.795386 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795484 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795484 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6tp\" (UniqueName: \"kubernetes.io/projected/7196b52a-4961-455d-9144-1795e9eb0dd7-kube-api-access-nd6tp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795724 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7196b52a-4961-455d-9144-1795e9eb0dd7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795724 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.795724 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.795684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.808624 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.808598 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5"] Jul 09 16:21:41.896055 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7196b52a-4961-455d-9144-1795e9eb0dd7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896055 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896304 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896304 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896304 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896304 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896498 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896498 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896498 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6tp\" (UniqueName: \"kubernetes.io/projected/7196b52a-4961-455d-9144-1795e9eb0dd7-kube-api-access-nd6tp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896498 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896486 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896686 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896686 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896812 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.896961 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.896946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7196b52a-4961-455d-9144-1795e9eb0dd7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.898443 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.898424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.898732 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.898711 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.909977 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.909938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6tp\" (UniqueName: \"kubernetes.io/projected/7196b52a-4961-455d-9144-1795e9eb0dd7-kube-api-access-nd6tp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:41.910561 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:41.910542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7196b52a-4961-455d-9144-1795e9eb0dd7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678jcbx5\" (UID: \"7196b52a-4961-455d-9144-1795e9eb0dd7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:42.014260 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.014222 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj"] Jul 09 16:21:42.018755 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.018734 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.022477 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.022450 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 16:21:42.023290 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.023273 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xwwj4\"" Jul 09 16:21:42.023408 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.023388 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 16:21:42.023516 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.023499 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 16:21:42.062553 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.062464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj"] Jul 09 16:21:42.096329 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.096293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:42.097636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.097707 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.097707 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwh8k\" (UniqueName: \"kubernetes.io/projected/4018af85-7af5-4424-8b76-b74aa59c4b82-kube-api-access-qwh8k\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.097794 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.097794 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.097909 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.097909 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.097870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018af85-7af5-4424-8b76-b74aa59c4b82-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.198660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.198660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.198660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.199030 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018af85-7af5-4424-8b76-b74aa59c4b82-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.199030 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.199030 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.199030 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.198805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwh8k\" (UniqueName: \"kubernetes.io/projected/4018af85-7af5-4424-8b76-b74aa59c4b82-kube-api-access-qwh8k\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.201273 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.199982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.202656 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.202389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.202656 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.202604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018af85-7af5-4424-8b76-b74aa59c4b82-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.202834 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.202758 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.203733 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.203704 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4018af85-7af5-4424-8b76-b74aa59c4b82-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.215862 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.215835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018af85-7af5-4424-8b76-b74aa59c4b82-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.218107 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.218085 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwh8k\" (UniqueName: \"kubernetes.io/projected/4018af85-7af5-4424-8b76-b74aa59c4b82-kube-api-access-qwh8k\") pod \"istiod-openshift-gateway-94bb8fbfd-n7lbj\" (UID: \"4018af85-7af5-4424-8b76-b74aa59c4b82\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.246217 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.246192 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5"] Jul 09 16:21:42.248486 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:21:42.248453 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7196b52a_4961_455d_9144_1795e9eb0dd7.slice/crio-a16c8780ac91a3f18316572ab6303657803a0ca697a47363a27d3a0b700513df WatchSource:0}: Error finding container a16c8780ac91a3f18316572ab6303657803a0ca697a47363a27d3a0b700513df: Status 404 returned error can't find the container with id a16c8780ac91a3f18316572ab6303657803a0ca697a47363a27d3a0b700513df Jul 09 16:21:42.322078 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.321987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" event={"ID":"7196b52a-4961-455d-9144-1795e9eb0dd7","Type":"ContainerStarted","Data":"a16c8780ac91a3f18316572ab6303657803a0ca697a47363a27d3a0b700513df"} Jul 09 16:21:42.328294 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.328270 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:42.485089 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:42.485059 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj"] Jul 09 16:21:42.487521 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:21:42.487489 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4018af85_7af5_4424_8b76_b74aa59c4b82.slice/crio-36dd8bb5629eb9d64fb7798aa1033660375f489e96360db4009ef2c66ce8825c WatchSource:0}: Error finding container 36dd8bb5629eb9d64fb7798aa1033660375f489e96360db4009ef2c66ce8825c: Status 404 returned error can't find the container with id 36dd8bb5629eb9d64fb7798aa1033660375f489e96360db4009ef2c66ce8825c Jul 09 16:21:43.328793 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:43.328747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" event={"ID":"4018af85-7af5-4424-8b76-b74aa59c4b82","Type":"ContainerStarted","Data":"36dd8bb5629eb9d64fb7798aa1033660375f489e96360db4009ef2c66ce8825c"} Jul 09 16:21:45.069782 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.069744 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:21:45.070170 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.069826 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:21:45.343215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.343126 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" event={"ID":"4018af85-7af5-4424-8b76-b74aa59c4b82","Type":"ContainerStarted","Data":"f431b23df95f17c0643a1a33f7775ebc855de11e2db5810bec34b7228a3d1f6f"} Jul 09 16:21:45.343342 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.343247 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:45.394013 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.393956 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" podStartSLOduration=1.814528833 podStartE2EDuration="4.393935165s" totalImagesPullingTime="2.579406332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:45.391810297 +0000 UTC m=+383.190030882" watchObservedRunningTime="2026-07-09 16:21:45.393935165 +0000 UTC m=+383.192155708" Jul 09 16:21:45.545162 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.545124 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq"] Jul 09 16:21:45.548755 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.548719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:45.556581 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.556555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bfn9w\"" Jul 09 16:21:45.556792 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.556607 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 16:21:45.597134 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.597034 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn"] Jul 09 16:21:45.601390 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.601361 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:45.604546 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.604513 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 16:21:45.605019 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.604996 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sbrkc\"" Jul 09 16:21:45.625723 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.625669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/647bf3ec-295b-4e39-b3c8-f98c2593d541-cert\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:45.625897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.625767 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqc4n\" (UniqueName: \"kubernetes.io/projected/8d6c4681-bee6-4871-82e0-8642bfba1482-kube-api-access-tqc4n\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:45.625897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.625844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c4681-bee6-4871-82e0-8642bfba1482-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:45.625995 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.625921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767ng\" (UniqueName: \"kubernetes.io/projected/647bf3ec-295b-4e39-b3c8-f98c2593d541-kube-api-access-767ng\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:45.649616 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.649578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq"] Jul 09 16:21:45.682992 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.682955 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn"] Jul 09 16:21:45.727262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.727223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/647bf3ec-295b-4e39-b3c8-f98c2593d541-cert\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:45.727460 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.727279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqc4n\" (UniqueName: \"kubernetes.io/projected/8d6c4681-bee6-4871-82e0-8642bfba1482-kube-api-access-tqc4n\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:45.727460 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.727307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c4681-bee6-4871-82e0-8642bfba1482-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:45.727460 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.727345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767ng\" (UniqueName: \"kubernetes.io/projected/647bf3ec-295b-4e39-b3c8-f98c2593d541-kube-api-access-767ng\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:45.727804 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:21:45.727781 2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 09 16:21:45.727868 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:21:45.727853 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/647bf3ec-295b-4e39-b3c8-f98c2593d541-cert podName:647bf3ec-295b-4e39-b3c8-f98c2593d541 nodeName:}" failed. No retries permitted until 2026-07-09 16:21:46.227832181 +0000 UTC m=+384.026052706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/647bf3ec-295b-4e39-b3c8-f98c2593d541-cert") pod "kserve-controller-manager-d6f4b65b8-mqgzq" (UID: "647bf3ec-295b-4e39-b3c8-f98c2593d541") : secret "kserve-webhook-server-cert" not found Jul 09 16:21:45.728137 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:21:45.728121 2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 09 16:21:45.728191 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:21:45.728171 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d6c4681-bee6-4871-82e0-8642bfba1482-cert podName:8d6c4681-bee6-4871-82e0-8642bfba1482 nodeName:}" failed. No retries permitted until 2026-07-09 16:21:46.228158669 +0000 UTC m=+384.026379193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8d6c4681-bee6-4871-82e0-8642bfba1482-cert") pod "llmisvc-controller-manager-777c6bd7d5-zt9cn" (UID: "8d6c4681-bee6-4871-82e0-8642bfba1482") : secret "llmisvc-webhook-server-cert" not found Jul 09 16:21:45.741325 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.741293 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqc4n\" (UniqueName: \"kubernetes.io/projected/8d6c4681-bee6-4871-82e0-8642bfba1482-kube-api-access-tqc4n\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:45.741704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:45.741686 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767ng\" (UniqueName: \"kubernetes.io/projected/647bf3ec-295b-4e39-b3c8-f98c2593d541-kube-api-access-767ng\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:46.231481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.231441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/647bf3ec-295b-4e39-b3c8-f98c2593d541-cert\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:46.231899 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.231490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c4681-bee6-4871-82e0-8642bfba1482-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:46.233849 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.233825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d6c4681-bee6-4871-82e0-8642bfba1482-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zt9cn\" (UID: \"8d6c4681-bee6-4871-82e0-8642bfba1482\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:46.233983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.233868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/647bf3ec-295b-4e39-b3c8-f98c2593d541-cert\") pod \"kserve-controller-manager-d6f4b65b8-mqgzq\" (UID: \"647bf3ec-295b-4e39-b3c8-f98c2593d541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:46.350376 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.350350 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-n7lbj" Jul 09 16:21:46.462509 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.462461 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:46.515101 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.515042 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:46.611603 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.611117 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq"] Jul 09 16:21:46.614034 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:21:46.614002 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647bf3ec_295b_4e39_b3c8_f98c2593d541.slice/crio-a1733450f63e1bdc4e5ca5c9b83f50c47346f110a41cdfd25b1b81adc7110aa9 WatchSource:0}: Error finding container a1733450f63e1bdc4e5ca5c9b83f50c47346f110a41cdfd25b1b81adc7110aa9: Status 404 returned error can't find the container with id a1733450f63e1bdc4e5ca5c9b83f50c47346f110a41cdfd25b1b81adc7110aa9 Jul 09 16:21:46.676507 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:46.676480 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn"] Jul 09 16:21:46.678582 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:21:46.678547 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d6c4681_bee6_4871_82e0_8642bfba1482.slice/crio-cd8eef79a66b1489d34595f47c6c416c6aac0704b69d8f566567502b1d797bae WatchSource:0}: Error finding container cd8eef79a66b1489d34595f47c6c416c6aac0704b69d8f566567502b1d797bae: Status 404 returned error can't find the container with id cd8eef79a66b1489d34595f47c6c416c6aac0704b69d8f566567502b1d797bae Jul 09 16:21:47.353927 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:47.353868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" event={"ID":"8d6c4681-bee6-4871-82e0-8642bfba1482","Type":"ContainerStarted","Data":"cd8eef79a66b1489d34595f47c6c416c6aac0704b69d8f566567502b1d797bae"} Jul 09 16:21:47.355036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:47.355010 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" event={"ID":"647bf3ec-295b-4e39-b3c8-f98c2593d541","Type":"ContainerStarted","Data":"a1733450f63e1bdc4e5ca5c9b83f50c47346f110a41cdfd25b1b81adc7110aa9"} Jul 09 16:21:50.373701 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:50.373655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" event={"ID":"647bf3ec-295b-4e39-b3c8-f98c2593d541","Type":"ContainerStarted","Data":"f938ddc02f7e92c4711d9c54bfba47dc1d780b2fb9d0bafa95b80730e29416c2"} Jul 09 16:21:50.374226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:50.373784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:21:50.422983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:50.422900 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" podStartSLOduration=2.2798574 podStartE2EDuration="5.422865002s" totalImagesPullingTime="3.143007602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:50.421427236 +0000 UTC m=+388.219647779" watchObservedRunningTime="2026-07-09 16:21:50.422865002 +0000 UTC m=+388.221085546" Jul 09 16:21:51.380597 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:51.380551 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" event={"ID":"8d6c4681-bee6-4871-82e0-8642bfba1482","Type":"ContainerStarted","Data":"6fb34dfbf8833cdf4542b3baac43fb1e72ff6f7037d4edd486265cbf6fab7952"} Jul 09 16:21:51.381139 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:51.380651 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:21:51.412558 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:51.412500 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" podStartSLOduration=2.450445336 podStartE2EDuration="6.412481633s" totalImagesPullingTime="3.962036297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:51.410249675 +0000 UTC m=+389.208470230" watchObservedRunningTime="2026-07-09 16:21:51.412481633 +0000 UTC m=+389.210702175" Jul 09 16:21:51.823309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:51.823273 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:21:51.823424 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:51.823345 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:21:51.823424 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:51.823372 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:21:52.385875 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:52.385837 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" event={"ID":"7196b52a-4961-455d-9144-1795e9eb0dd7","Type":"ContainerStarted","Data":"1125f11f7fabe4b5f086dd4759bf373935bc8404873786302097ee1a1c476d59"} Jul 09 16:21:52.409737 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:52.409683 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" podStartSLOduration=1.837354739 podStartE2EDuration="11.409665104s" totalImagesPullingTime="9.572310365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:52.408677634 +0000 UTC m=+390.206898177" watchObservedRunningTime="2026-07-09 16:21:52.409665104 +0000 UTC m=+390.207885667" Jul 09 16:21:53.097403 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:53.097364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:53.102577 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:53.102552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:53.390185 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:53.390103 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:21:53.391229 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:21:53.391208 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678jcbx5" Jul 09 16:22:21.386036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:22:21.386003 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mqgzq" Jul 09 16:22:22.388782 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:22:22.388753 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zt9cn" Jul 09 16:23:07.678581 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.678548 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-77sd6"] Jul 09 16:23:07.682020 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.682003 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:07.685039 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.685021 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 16:23:07.686568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.686549 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 16:23:07.686654 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.686620 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fbrgf\"" Jul 09 16:23:07.694994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.694970 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-77sd6"] Jul 09 16:23:07.843953 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.843923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kd4x\" (UniqueName: \"kubernetes.io/projected/c505acff-5ef8-4bd7-9d7b-522e27721902-kube-api-access-8kd4x\") pod \"authorino-operator-75cc8d5cb7-77sd6\" (UID: \"c505acff-5ef8-4bd7-9d7b-522e27721902\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:07.944692 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.944606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kd4x\" (UniqueName: \"kubernetes.io/projected/c505acff-5ef8-4bd7-9d7b-522e27721902-kube-api-access-8kd4x\") pod \"authorino-operator-75cc8d5cb7-77sd6\" (UID: \"c505acff-5ef8-4bd7-9d7b-522e27721902\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:07.965139 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.965115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kd4x\" (UniqueName: \"kubernetes.io/projected/c505acff-5ef8-4bd7-9d7b-522e27721902-kube-api-access-8kd4x\") pod \"authorino-operator-75cc8d5cb7-77sd6\" (UID: \"c505acff-5ef8-4bd7-9d7b-522e27721902\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:07.991613 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:07.991588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:08.122995 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:08.122967 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-77sd6"] Jul 09 16:23:08.125632 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:23:08.125603 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc505acff_5ef8_4bd7_9d7b_522e27721902.slice/crio-3316bc7456b6434c0b93dea63664e9810082747d464fd818bf8ee9c7dea279f7 WatchSource:0}: Error finding container 3316bc7456b6434c0b93dea63664e9810082747d464fd818bf8ee9c7dea279f7: Status 404 returned error can't find the container with id 3316bc7456b6434c0b93dea63664e9810082747d464fd818bf8ee9c7dea279f7 Jul 09 16:23:08.713051 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:08.713023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" event={"ID":"c505acff-5ef8-4bd7-9d7b-522e27721902","Type":"ContainerStarted","Data":"3316bc7456b6434c0b93dea63664e9810082747d464fd818bf8ee9c7dea279f7"} Jul 09 16:23:12.734533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:12.734500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" event={"ID":"c505acff-5ef8-4bd7-9d7b-522e27721902","Type":"ContainerStarted","Data":"b23207f3c7ba17b88bdf9f53ab9d7990ce222ee3ea0d69544032fa47317d4d3c"} Jul 09 16:23:12.734935 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:12.734654 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:12.759226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:12.759182 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" podStartSLOduration=1.272691512 podStartE2EDuration="5.759167999s" totalImagesPullingTime="4.486476487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:12.758295963 +0000 UTC m=+470.556516504" watchObservedRunningTime="2026-07-09 16:23:12.759167999 +0000 UTC m=+470.557388541" Jul 09 16:23:23.742525 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:23.742498 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-77sd6" Jul 09 16:23:36.696834 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.696799 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf"] Jul 09 16:23:36.700196 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.700176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:36.705013 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.704988 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p94jf\"" Jul 09 16:23:36.732013 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.731992 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf"] Jul 09 16:23:36.803061 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.803035 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4522430-3ab7-4261-98f6-3b3ea04de3aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:36.803203 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.803074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmk8\" (UniqueName: \"kubernetes.io/projected/a4522430-3ab7-4261-98f6-3b3ea04de3aa-kube-api-access-hcmk8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:36.904427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.904396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4522430-3ab7-4261-98f6-3b3ea04de3aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:36.904427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.904430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmk8\" (UniqueName: \"kubernetes.io/projected/a4522430-3ab7-4261-98f6-3b3ea04de3aa-kube-api-access-hcmk8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:36.904803 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.904781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4522430-3ab7-4261-98f6-3b3ea04de3aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:36.915407 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:36.915387 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmk8\" (UniqueName: \"kubernetes.io/projected/a4522430-3ab7-4261-98f6-3b3ea04de3aa-kube-api-access-hcmk8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:37.009757 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.009689 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:37.142200 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.142176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf"] Jul 09 16:23:37.144749 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:23:37.144719 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4522430_3ab7_4261_98f6_3b3ea04de3aa.slice/crio-a9c16eaee04b3d7929e82bdbcf6b91de1366a0fc0ac5b6c242876c07e30f17e8 WatchSource:0}: Error finding container a9c16eaee04b3d7929e82bdbcf6b91de1366a0fc0ac5b6c242876c07e30f17e8: Status 404 returned error can't find the container with id a9c16eaee04b3d7929e82bdbcf6b91de1366a0fc0ac5b6c242876c07e30f17e8 Jul 09 16:23:37.457150 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.457122 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf"] Jul 09 16:23:37.461058 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.461031 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf"] Jul 09 16:23:37.490058 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.490031 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh"] Jul 09 16:23:37.495777 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.495743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.516604 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.516575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh"] Jul 09 16:23:37.608957 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.608925 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhvs\" (UniqueName: \"kubernetes.io/projected/dc19bbb1-70cc-4110-bf36-d822adfce55a-kube-api-access-knhvs\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gx7mh\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.609137 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.608981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc19bbb1-70cc-4110-bf36-d822adfce55a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gx7mh\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.709540 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.709459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhvs\" (UniqueName: \"kubernetes.io/projected/dc19bbb1-70cc-4110-bf36-d822adfce55a-kube-api-access-knhvs\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gx7mh\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.709540 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.709521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc19bbb1-70cc-4110-bf36-d822adfce55a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gx7mh\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.709977 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.709957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc19bbb1-70cc-4110-bf36-d822adfce55a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gx7mh\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.722059 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.722034 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhvs\" (UniqueName: \"kubernetes.io/projected/dc19bbb1-70cc-4110-bf36-d822adfce55a-kube-api-access-knhvs\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gx7mh\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:37.809343 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:37.809304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:38.256526 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:38.256493 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh"] Jul 09 16:23:38.288472 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:23:38.288436 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc19bbb1_70cc_4110_bf36_d822adfce55a.slice/crio-79de297071692a6b60af0479182fa24551deb81d6b865b4d867cdfce64f18ab6 WatchSource:0}: Error finding container 79de297071692a6b60af0479182fa24551deb81d6b865b4d867cdfce64f18ab6: Status 404 returned error can't find the container with id 79de297071692a6b60af0479182fa24551deb81d6b865b4d867cdfce64f18ab6 Jul 09 16:23:38.854351 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:38.854315 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" event={"ID":"dc19bbb1-70cc-4110-bf36-d822adfce55a","Type":"ContainerStarted","Data":"79de297071692a6b60af0479182fa24551deb81d6b865b4d867cdfce64f18ab6"} Jul 09 16:23:42.874152 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:42.874081 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" containerName="manager" containerID="cri-o://e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55" gracePeriod=2 Jul 09 16:23:42.875697 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:42.875671 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" event={"ID":"dc19bbb1-70cc-4110-bf36-d822adfce55a","Type":"ContainerStarted","Data":"e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10"} Jul 09 16:23:42.875828 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:42.875811 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:42.876658 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:23:42.876632 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:42.879106 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:23:42.879082 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:42.900683 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:42.900644 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" podStartSLOduration=1.504348651 podStartE2EDuration="5.900631593s" totalImagesPullingTime="4.396282942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:42.900084418 +0000 UTC m=+500.698304959" watchObservedRunningTime="2026-07-09 16:23:42.900631593 +0000 UTC m=+500.698852134" Jul 09 16:23:43.112016 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.111993 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:43.117403 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:23:43.117373 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:43.261856 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.261764 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4522430-3ab7-4261-98f6-3b3ea04de3aa-extensions-socket-volume\") pod \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " Jul 09 16:23:43.261856 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.261805 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcmk8\" (UniqueName: \"kubernetes.io/projected/a4522430-3ab7-4261-98f6-3b3ea04de3aa-kube-api-access-hcmk8\") pod \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\" (UID: \"a4522430-3ab7-4261-98f6-3b3ea04de3aa\") " Jul 09 16:23:43.262107 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.262056 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4522430-3ab7-4261-98f6-3b3ea04de3aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a4522430-3ab7-4261-98f6-3b3ea04de3aa" (UID: "a4522430-3ab7-4261-98f6-3b3ea04de3aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:23:43.263955 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.263933 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4522430-3ab7-4261-98f6-3b3ea04de3aa-kube-api-access-hcmk8" (OuterVolumeSpecName: "kube-api-access-hcmk8") pod "a4522430-3ab7-4261-98f6-3b3ea04de3aa" (UID: "a4522430-3ab7-4261-98f6-3b3ea04de3aa"). InnerVolumeSpecName "kube-api-access-hcmk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:23:43.363301 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.363253 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4522430-3ab7-4261-98f6-3b3ea04de3aa-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:23:43.363301 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.363292 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmk8\" (UniqueName: \"kubernetes.io/projected/a4522430-3ab7-4261-98f6-3b3ea04de3aa-kube-api-access-hcmk8\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:23:43.880928 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.880893 2562 generic.go:356] "Generic (PLEG): container finished" podID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" containerID="e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55" exitCode=2 Jul 09 16:23:43.881343 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.880937 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:43.881343 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.880987 2562 scope.go:117] "RemoveContainer" containerID="e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55" Jul 09 16:23:43.883727 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:23:43.883699 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:43.889818 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.889798 2562 scope.go:117] "RemoveContainer" containerID="e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55" Jul 09 16:23:43.890102 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:23:43.890084 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55\": container with ID starting with e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55 not found: ID does not exist" containerID="e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55" Jul 09 16:23:43.890145 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:43.890111 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55"} err="failed to get container status \"e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55\": rpc error: code = NotFound desc = could not find container \"e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55\": container with ID starting with e3d9ce56fe3b9da060a9e17cd420587e7f1b7b4d6b4ca5514a39f09268599c55 not found: ID does not exist" Jul 09 16:23:43.892159 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:23:43.892139 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2k6bf\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2k6bf" Jul 09 16:23:44.821108 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:44.821074 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" path="/var/lib/kubelet/pods/a4522430-3ab7-4261-98f6-3b3ea04de3aa/volumes" Jul 09 16:23:53.883587 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:53.883557 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:23:54.225049 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.225012 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq"] Jul 09 16:23:54.225352 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.225339 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" containerName="manager" Jul 09 16:23:54.225394 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.225353 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" containerName="manager" Jul 09 16:23:54.225426 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.225414 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4522430-3ab7-4261-98f6-3b3ea04de3aa" containerName="manager" Jul 09 16:23:54.228281 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.228265 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.250627 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.250599 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq"] Jul 09 16:23:54.267424 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.267396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad116c02-b8fe-463f-befe-e3d6dc075a09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6d9xq\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.267589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.267434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/ad116c02-b8fe-463f-befe-e3d6dc075a09-kube-api-access-rwhp2\") pod \"kuadrant-operator-controller-manager-86878c769f-6d9xq\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.368128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.368083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad116c02-b8fe-463f-befe-e3d6dc075a09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6d9xq\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.368128 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.368131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/ad116c02-b8fe-463f-befe-e3d6dc075a09-kube-api-access-rwhp2\") pod \"kuadrant-operator-controller-manager-86878c769f-6d9xq\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.368495 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.368474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad116c02-b8fe-463f-befe-e3d6dc075a09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6d9xq\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.378175 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.378140 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/ad116c02-b8fe-463f-befe-e3d6dc075a09-kube-api-access-rwhp2\") pod \"kuadrant-operator-controller-manager-86878c769f-6d9xq\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.537712 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.537627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.668678 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.668654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq"] Jul 09 16:23:54.671288 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:23:54.671257 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad116c02_b8fe_463f_befe_e3d6dc075a09.slice/crio-4784570efea6e02819d56c653f84478c6ad83a1930de575013b78fb64df96a34 WatchSource:0}: Error finding container 4784570efea6e02819d56c653f84478c6ad83a1930de575013b78fb64df96a34: Status 404 returned error can't find the container with id 4784570efea6e02819d56c653f84478c6ad83a1930de575013b78fb64df96a34 Jul 09 16:23:54.931005 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.930964 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" event={"ID":"ad116c02-b8fe-463f-befe-e3d6dc075a09","Type":"ContainerStarted","Data":"e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff"} Jul 09 16:23:54.931494 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.931023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" event={"ID":"ad116c02-b8fe-463f-befe-e3d6dc075a09","Type":"ContainerStarted","Data":"4784570efea6e02819d56c653f84478c6ad83a1930de575013b78fb64df96a34"} Jul 09 16:23:54.931494 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.931161 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:23:54.954283 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:23:54.954229 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" podStartSLOduration=0.954213481 podStartE2EDuration="954.213481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:54.953452576 +0000 UTC m=+512.751673129" watchObservedRunningTime="2026-07-09 16:23:54.954213481 +0000 UTC m=+512.752434023" Jul 09 16:24:05.937738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:05.937708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:24:05.995165 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:05.995135 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh"] Jul 09 16:24:05.995377 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:05.995344 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" podUID="dc19bbb1-70cc-4110-bf36-d822adfce55a" containerName="manager" containerID="cri-o://e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10" gracePeriod=10 Jul 09 16:24:06.237754 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.237728 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:24:06.273994 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.273964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knhvs\" (UniqueName: \"kubernetes.io/projected/dc19bbb1-70cc-4110-bf36-d822adfce55a-kube-api-access-knhvs\") pod \"dc19bbb1-70cc-4110-bf36-d822adfce55a\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " Jul 09 16:24:06.274177 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.274144 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc19bbb1-70cc-4110-bf36-d822adfce55a-extensions-socket-volume\") pod \"dc19bbb1-70cc-4110-bf36-d822adfce55a\" (UID: \"dc19bbb1-70cc-4110-bf36-d822adfce55a\") " Jul 09 16:24:06.274618 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.274588 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc19bbb1-70cc-4110-bf36-d822adfce55a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dc19bbb1-70cc-4110-bf36-d822adfce55a" (UID: "dc19bbb1-70cc-4110-bf36-d822adfce55a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:24:06.276037 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.276017 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc19bbb1-70cc-4110-bf36-d822adfce55a-kube-api-access-knhvs" (OuterVolumeSpecName: "kube-api-access-knhvs") pod "dc19bbb1-70cc-4110-bf36-d822adfce55a" (UID: "dc19bbb1-70cc-4110-bf36-d822adfce55a"). InnerVolumeSpecName "kube-api-access-knhvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:24:06.375551 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.375521 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc19bbb1-70cc-4110-bf36-d822adfce55a-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:24:06.375551 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.375548 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knhvs\" (UniqueName: \"kubernetes.io/projected/dc19bbb1-70cc-4110-bf36-d822adfce55a-kube-api-access-knhvs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:24:06.983271 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.983235 2562 generic.go:356] "Generic (PLEG): container finished" podID="dc19bbb1-70cc-4110-bf36-d822adfce55a" containerID="e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10" exitCode=0 Jul 09 16:24:06.983720 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.983283 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" Jul 09 16:24:06.983720 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.983321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" event={"ID":"dc19bbb1-70cc-4110-bf36-d822adfce55a","Type":"ContainerDied","Data":"e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10"} Jul 09 16:24:06.983720 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.983381 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh" event={"ID":"dc19bbb1-70cc-4110-bf36-d822adfce55a","Type":"ContainerDied","Data":"79de297071692a6b60af0479182fa24551deb81d6b865b4d867cdfce64f18ab6"} Jul 09 16:24:06.983720 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.983413 2562 scope.go:117] "RemoveContainer" containerID="e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10" Jul 09 16:24:06.991460 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.991445 2562 scope.go:117] "RemoveContainer" containerID="e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10" Jul 09 16:24:06.991679 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:24:06.991661 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10\": container with ID starting with e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10 not found: ID does not exist" containerID="e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10" Jul 09 16:24:06.991728 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:06.991688 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10"} err="failed to get container status \"e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10\": rpc error: code = NotFound desc = could not find container \"e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10\": container with ID starting with e911da1c3136aedb952260d75d294855be695458af0e026d5dc57947ce492f10 not found: ID does not exist" Jul 09 16:24:07.012010 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:07.011987 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh"] Jul 09 16:24:07.018653 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:07.018633 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gx7mh"] Jul 09 16:24:08.819461 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:08.819426 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc19bbb1-70cc-4110-bf36-d822adfce55a" path="/var/lib/kubelet/pods/dc19bbb1-70cc-4110-bf36-d822adfce55a/volumes" Jul 09 16:24:10.379955 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.379921 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2"] Jul 09 16:24:10.380422 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.380284 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc19bbb1-70cc-4110-bf36-d822adfce55a" containerName="manager" Jul 09 16:24:10.380422 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.380297 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc19bbb1-70cc-4110-bf36-d822adfce55a" containerName="manager" Jul 09 16:24:10.380422 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.380351 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc19bbb1-70cc-4110-bf36-d822adfce55a" containerName="manager" Jul 09 16:24:10.386253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.386233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.392364 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.392338 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b547v\"" Jul 09 16:24:10.399101 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.399071 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2"] Jul 09 16:24:10.512273 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512273 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512496 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512496 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512496 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512496 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b10fa6-eb53-417a-8f52-79b9407df75d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512496 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512676 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wmfr\" (UniqueName: \"kubernetes.io/projected/25b10fa6-eb53-417a-8f52-79b9407df75d-kube-api-access-8wmfr\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.512676 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.512562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.613751 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.613717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.613974 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.613785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.613974 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.613813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.613974 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.613834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b10fa6-eb53-417a-8f52-79b9407df75d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.613974 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.613899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614175 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wmfr\" (UniqueName: \"kubernetes.io/projected/25b10fa6-eb53-417a-8f52-79b9407df75d-kube-api-access-8wmfr\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614175 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614175 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614348 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614348 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614255 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614348 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614652 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614672 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.614846 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.614826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b10fa6-eb53-417a-8f52-79b9407df75d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.616728 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.616704 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.616836 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.616736 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.624874 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.624851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b10fa6-eb53-417a-8f52-79b9407df75d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.625156 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.625133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wmfr\" (UniqueName: \"kubernetes.io/projected/25b10fa6-eb53-417a-8f52-79b9407df75d-kube-api-access-8wmfr\") pod \"maas-default-gateway-openshift-default-5896d4dd49-btwv2\" (UID: \"25b10fa6-eb53-417a-8f52-79b9407df75d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.701861 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.701829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:10.840329 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.840304 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2"] Jul 09 16:24:10.842631 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:24:10.842605 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b10fa6_eb53_417a_8f52_79b9407df75d.slice/crio-34443a17f96b7f6f7a842f3fffd2217e94fbe6d6382323b49bb183294243fdc1 WatchSource:0}: Error finding container 34443a17f96b7f6f7a842f3fffd2217e94fbe6d6382323b49bb183294243fdc1: Status 404 returned error can't find the container with id 34443a17f96b7f6f7a842f3fffd2217e94fbe6d6382323b49bb183294243fdc1 Jul 09 16:24:10.845094 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.845064 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:24:10.845181 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.845140 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:24:10.845246 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:10.845182 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:24:11.004140 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:11.004052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" event={"ID":"25b10fa6-eb53-417a-8f52-79b9407df75d","Type":"ContainerStarted","Data":"031f1f33e5169a60256f0d1b949a021ab6d9e3e8f8771e36554ddfc69e8dbbe7"} Jul 09 16:24:11.004140 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:11.004117 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" event={"ID":"25b10fa6-eb53-417a-8f52-79b9407df75d","Type":"ContainerStarted","Data":"34443a17f96b7f6f7a842f3fffd2217e94fbe6d6382323b49bb183294243fdc1"} Jul 09 16:24:11.026195 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:11.026124 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" podStartSLOduration=1.026105279 podStartE2EDuration="1.026105279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:11.024178437 +0000 UTC m=+528.822398983" watchObservedRunningTime="2026-07-09 16:24:11.026105279 +0000 UTC m=+528.824325821" Jul 09 16:24:11.702941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:11.702904 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:11.707968 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:11.707945 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:12.008783 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:12.008701 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:24:12.009675 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:24:12.009658 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:25:22.682394 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:22.682369 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:25:22.683275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:22.683255 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:25:38.396027 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:38.395994 2562 generic.go:356] "Generic (PLEG): container finished" podID="25b10fa6-eb53-417a-8f52-79b9407df75d" containerID="031f1f33e5169a60256f0d1b949a021ab6d9e3e8f8771e36554ddfc69e8dbbe7" exitCode=0 Jul 09 16:25:38.396454 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:38.396058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" event={"ID":"25b10fa6-eb53-417a-8f52-79b9407df75d","Type":"ContainerDied","Data":"031f1f33e5169a60256f0d1b949a021ab6d9e3e8f8771e36554ddfc69e8dbbe7"} Jul 09 16:25:38.396454 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:38.396445 2562 scope.go:117] "RemoveContainer" containerID="031f1f33e5169a60256f0d1b949a021ab6d9e3e8f8771e36554ddfc69e8dbbe7" Jul 09 16:25:38.397462 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:38.397435 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:25:38.397537 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:38.397520 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:25:38.397590 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:38.397574 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:25:39.402287 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:39.402250 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" event={"ID":"25b10fa6-eb53-417a-8f52-79b9407df75d","Type":"ContainerStarted","Data":"765126bfd96828211938aea5cba2643350afbd7437aa7e2dfe9bb11384acf1e4"} Jul 09 16:25:39.702232 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:39.702197 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:25:39.703382 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:39.703355 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-btwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 09 16:25:39.703514 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:39.703410 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" podUID="25b10fa6-eb53-417a-8f52-79b9407df75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 09 16:25:40.702426 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:40.702390 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-btwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 09 16:25:40.702829 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:40.702442 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" podUID="25b10fa6-eb53-417a-8f52-79b9407df75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 09 16:25:41.702615 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:41.702580 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-btwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 09 16:25:41.703019 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:41.702636 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" podUID="25b10fa6-eb53-417a-8f52-79b9407df75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 09 16:25:42.009693 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:42.009606 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:25:42.702660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:42.702623 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-btwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 09 16:25:42.703114 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:42.702693 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" podUID="25b10fa6-eb53-417a-8f52-79b9407df75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 09 16:25:43.703083 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:43.703041 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-btwv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 09 16:25:43.703463 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:43.703109 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" podUID="25b10fa6-eb53-417a-8f52-79b9407df75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 09 16:25:44.707108 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:44.707080 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:25:44.707950 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:44.707933 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-btwv2" Jul 09 16:25:49.683533 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.683499 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5"] Jul 09 16:25:49.686913 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.686872 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.690068 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.690046 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 16:25:49.691206 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.691185 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 16:25:49.691326 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.691219 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 16:25:49.691326 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.691298 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8g5kt\"" Jul 09 16:25:49.699080 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.699047 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5"] Jul 09 16:25:49.809505 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.809467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.809505 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.809507 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.809778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.809697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.809778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.809732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89bd19da-4752-45e7-b8ea-514196e34df0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.809778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.809760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.809943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.809784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86c2\" (UniqueName: \"kubernetes.io/projected/89bd19da-4752-45e7-b8ea-514196e34df0-kube-api-access-t86c2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.910975 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.910937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.910975 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.910975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89bd19da-4752-45e7-b8ea-514196e34df0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911257 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.910998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911257 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.911021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86c2\" (UniqueName: \"kubernetes.io/projected/89bd19da-4752-45e7-b8ea-514196e34df0-kube-api-access-t86c2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911257 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.911056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911257 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.911092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911578 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.911552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911721 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.911554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.911721 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.911591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.913462 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.913425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89bd19da-4752-45e7-b8ea-514196e34df0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.913605 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.913589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89bd19da-4752-45e7-b8ea-514196e34df0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.919310 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.919283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86c2\" (UniqueName: \"kubernetes.io/projected/89bd19da-4752-45e7-b8ea-514196e34df0-kube-api-access-t86c2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5\" (UID: \"89bd19da-4752-45e7-b8ea-514196e34df0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:49.998380 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:49.998294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:50.141798 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:50.141758 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5"] Jul 09 16:25:50.144228 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:25:50.144194 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bd19da_4752_45e7_b8ea_514196e34df0.slice/crio-738e90de229300c373cdf415f8d27a4c8007c58334e01aac895559953dac2de7 WatchSource:0}: Error finding container 738e90de229300c373cdf415f8d27a4c8007c58334e01aac895559953dac2de7: Status 404 returned error can't find the container with id 738e90de229300c373cdf415f8d27a4c8007c58334e01aac895559953dac2de7 Jul 09 16:25:50.145744 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:50.145716 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:25:50.451516 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:50.451479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" event={"ID":"89bd19da-4752-45e7-b8ea-514196e34df0","Type":"ContainerStarted","Data":"738e90de229300c373cdf415f8d27a4c8007c58334e01aac895559953dac2de7"} Jul 09 16:25:51.462900 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:51.462840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" event={"ID":"89bd19da-4752-45e7-b8ea-514196e34df0","Type":"ContainerStarted","Data":"82803d98f0ec7cc3df5ebbfe2bbf22ce318260deab940999f73940d8ad2fe78b"} Jul 09 16:25:51.482320 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:51.482268 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" podStartSLOduration=1.321052171 podStartE2EDuration="2.4822524s" totalImagesPullingTime="1.161200229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:25:51.480623724 +0000 UTC m=+629.278844267" watchObservedRunningTime="2026-07-09 16:25:51.4822524 +0000 UTC m=+629.280472941" Jul 09 16:25:59.999000 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:59.998963 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:25:59.999500 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:25:59.999035 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:26:00.011232 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:00.011204 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:26:00.513957 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:00.513927 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5" Jul 09 16:26:41.802413 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.802377 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv"] Jul 09 16:26:41.806467 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.806450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:41.810564 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.810545 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 09 16:26:41.815762 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.815738 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv"] Jul 09 16:26:41.980236 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.980195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:41.980397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.980277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:41.980397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.980340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:41.980397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.980366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df96aa47-bd8f-4732-8039-760050e92e56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:41.980397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.980389 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fnj\" (UniqueName: \"kubernetes.io/projected/df96aa47-bd8f-4732-8039-760050e92e56-kube-api-access-z2fnj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:41.980579 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:41.980465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.081810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.081713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.081810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.081752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df96aa47-bd8f-4732-8039-760050e92e56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.081810 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.081775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fnj\" (UniqueName: \"kubernetes.io/projected/df96aa47-bd8f-4732-8039-760050e92e56-kube-api-access-z2fnj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.082135 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.081816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.082135 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.081971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.082135 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.082077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.082278 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.082181 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.082278 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.082218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.082366 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.082342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.084162 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.084129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df96aa47-bd8f-4732-8039-760050e92e56-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.084327 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.084311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df96aa47-bd8f-4732-8039-760050e92e56-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.091868 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.091847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fnj\" (UniqueName: \"kubernetes.io/projected/df96aa47-bd8f-4732-8039-760050e92e56-kube-api-access-z2fnj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6t9fv\" (UID: \"df96aa47-bd8f-4732-8039-760050e92e56\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.116744 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.116718 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:42.257759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.257737 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv"] Jul 09 16:26:42.259651 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:26:42.259616 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf96aa47_bd8f_4732_8039_760050e92e56.slice/crio-3a90e97cc87979b3d5242cf20d3395f86e42ccfe85575042846553f540b3fbec WatchSource:0}: Error finding container 3a90e97cc87979b3d5242cf20d3395f86e42ccfe85575042846553f540b3fbec: Status 404 returned error can't find the container with id 3a90e97cc87979b3d5242cf20d3395f86e42ccfe85575042846553f540b3fbec Jul 09 16:26:42.704789 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.704749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" event={"ID":"df96aa47-bd8f-4732-8039-760050e92e56","Type":"ContainerStarted","Data":"b950e84ac5e8aeeb26fc9e1cde7a68a86e7d4cba05d3fc76feb1f3e59bd811bb"} Jul 09 16:26:42.704962 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.704807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" event={"ID":"df96aa47-bd8f-4732-8039-760050e92e56","Type":"ContainerStarted","Data":"3a90e97cc87979b3d5242cf20d3395f86e42ccfe85575042846553f540b3fbec"} Jul 09 16:26:42.734344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:42.734295 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" podStartSLOduration=1.5025893240000001 podStartE2EDuration="1.734276915s" totalImagesPullingTime="231.687591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:26:42.732386599 +0000 UTC m=+680.530607197" watchObservedRunningTime="2026-07-09 16:26:42.734276915 +0000 UTC m=+680.532497457" Jul 09 16:26:52.117697 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:52.117645 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:52.118198 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:52.117721 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:52.132328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:52.132294 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:26:52.758172 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:26:52.758146 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6t9fv" Jul 09 16:30:22.708671 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:30:22.708587 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:30:22.710898 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:30:22.710856 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:33:35.646725 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:35.646645 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq"] Jul 09 16:33:35.647251 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:35.646829 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" podUID="ad116c02-b8fe-463f-befe-e3d6dc075a09" containerName="manager" containerID="cri-o://e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff" gracePeriod=10 Jul 09 16:33:38.905255 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:38.905231 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:33:39.053942 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.053852 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad116c02-b8fe-463f-befe-e3d6dc075a09-extensions-socket-volume\") pod \"ad116c02-b8fe-463f-befe-e3d6dc075a09\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " Jul 09 16:33:39.053942 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.053917 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/ad116c02-b8fe-463f-befe-e3d6dc075a09-kube-api-access-rwhp2\") pod \"ad116c02-b8fe-463f-befe-e3d6dc075a09\" (UID: \"ad116c02-b8fe-463f-befe-e3d6dc075a09\") " Jul 09 16:33:39.054355 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.054318 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad116c02-b8fe-463f-befe-e3d6dc075a09-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad116c02-b8fe-463f-befe-e3d6dc075a09" (UID: "ad116c02-b8fe-463f-befe-e3d6dc075a09"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:33:39.056018 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.055991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad116c02-b8fe-463f-befe-e3d6dc075a09-kube-api-access-rwhp2" (OuterVolumeSpecName: "kube-api-access-rwhp2") pod "ad116c02-b8fe-463f-befe-e3d6dc075a09" (UID: "ad116c02-b8fe-463f-befe-e3d6dc075a09"). InnerVolumeSpecName "kube-api-access-rwhp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:33:39.154930 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.154862 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad116c02-b8fe-463f-befe-e3d6dc075a09-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:33:39.154930 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.154924 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhp2\" (UniqueName: \"kubernetes.io/projected/ad116c02-b8fe-463f-befe-e3d6dc075a09-kube-api-access-rwhp2\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:33:39.614931 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.614872 2562 generic.go:356] "Generic (PLEG): container finished" podID="ad116c02-b8fe-463f-befe-e3d6dc075a09" containerID="e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff" exitCode=0 Jul 09 16:33:39.614931 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.614918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" event={"ID":"ad116c02-b8fe-463f-befe-e3d6dc075a09","Type":"ContainerDied","Data":"e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff"} Jul 09 16:33:39.615211 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.614949 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" Jul 09 16:33:39.615211 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.615018 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq" event={"ID":"ad116c02-b8fe-463f-befe-e3d6dc075a09","Type":"ContainerDied","Data":"4784570efea6e02819d56c653f84478c6ad83a1930de575013b78fb64df96a34"} Jul 09 16:33:39.615211 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.615050 2562 scope.go:117] "RemoveContainer" containerID="e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff" Jul 09 16:33:39.624583 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.624565 2562 scope.go:117] "RemoveContainer" containerID="e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff" Jul 09 16:33:39.624828 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:33:39.624809 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff\": container with ID starting with e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff not found: ID does not exist" containerID="e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff" Jul 09 16:33:39.624890 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.624837 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff"} err="failed to get container status \"e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff\": rpc error: code = NotFound desc = could not find container \"e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff\": container with ID starting with e0d25707fd54c9cf349ed55fde6ff9c208c964897ef88a1be70e92f8c282faff not found: ID does not exist" Jul 09 16:33:39.636894 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.636853 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq"] Jul 09 16:33:39.640246 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:39.640224 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6d9xq"] Jul 09 16:33:40.820639 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:33:40.820603 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad116c02-b8fe-463f-befe-e3d6dc075a09" path="/var/lib/kubelet/pods/ad116c02-b8fe-463f-befe-e3d6dc075a09/volumes" Jul 09 16:34:36.854169 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.854134 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk"] Jul 09 16:34:36.854572 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.854467 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad116c02-b8fe-463f-befe-e3d6dc075a09" containerName="manager" Jul 09 16:34:36.854572 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.854477 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad116c02-b8fe-463f-befe-e3d6dc075a09" containerName="manager" Jul 09 16:34:36.854572 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.854542 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad116c02-b8fe-463f-befe-e3d6dc075a09" containerName="manager" Jul 09 16:34:36.857767 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.857748 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:36.860857 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.860832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p94jf\"" Jul 09 16:34:36.869410 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.869385 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk"] Jul 09 16:34:36.875922 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.875897 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba7b15aa-e4cb-45b3-b3f6-210b14ed155d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x27xk\" (UID: \"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:36.876038 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.875944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mg4\" (UniqueName: \"kubernetes.io/projected/ba7b15aa-e4cb-45b3-b3f6-210b14ed155d-kube-api-access-q7mg4\") pod \"kuadrant-operator-controller-manager-86878c769f-x27xk\" (UID: \"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:36.976801 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.976770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba7b15aa-e4cb-45b3-b3f6-210b14ed155d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x27xk\" (UID: \"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:36.976983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.976808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mg4\" (UniqueName: \"kubernetes.io/projected/ba7b15aa-e4cb-45b3-b3f6-210b14ed155d-kube-api-access-q7mg4\") pod \"kuadrant-operator-controller-manager-86878c769f-x27xk\" (UID: \"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:36.977166 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.977142 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba7b15aa-e4cb-45b3-b3f6-210b14ed155d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x27xk\" (UID: \"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:36.989141 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:36.989110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mg4\" (UniqueName: \"kubernetes.io/projected/ba7b15aa-e4cb-45b3-b3f6-210b14ed155d-kube-api-access-q7mg4\") pod \"kuadrant-operator-controller-manager-86878c769f-x27xk\" (UID: \"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:37.168396 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:37.168368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:37.307870 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:37.307844 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk"] Jul 09 16:34:37.309704 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:34:37.309681 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7b15aa_e4cb_45b3_b3f6_210b14ed155d.slice/crio-ca5945838de816f9b29e3103a234f6dbcb8c646486476d07a6c089492e054b86 WatchSource:0}: Error finding container ca5945838de816f9b29e3103a234f6dbcb8c646486476d07a6c089492e054b86: Status 404 returned error can't find the container with id ca5945838de816f9b29e3103a234f6dbcb8c646486476d07a6c089492e054b86 Jul 09 16:34:37.874219 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:37.874186 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" event={"ID":"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d","Type":"ContainerStarted","Data":"dfeb671f6a4df85cb73b4da6b2f1d39cbb1b311a2b868815c76e72a16205e77d"} Jul 09 16:34:37.874605 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:37.874237 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" event={"ID":"ba7b15aa-e4cb-45b3-b3f6-210b14ed155d","Type":"ContainerStarted","Data":"ca5945838de816f9b29e3103a234f6dbcb8c646486476d07a6c089492e054b86"} Jul 09 16:34:37.874605 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:37.874310 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:34:37.900805 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:37.900761 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" podStartSLOduration=1.900745755 podStartE2EDuration="1.900745755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:37.898577194 +0000 UTC m=+1155.696797766" watchObservedRunningTime="2026-07-09 16:34:37.900745755 +0000 UTC m=+1155.698966296" Jul 09 16:34:48.881702 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:34:48.881669 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x27xk" Jul 09 16:35:22.740211 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:35:22.740167 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:35:22.744037 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:35:22.744011 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:40:22.777891 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:40:22.777862 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:40:22.781342 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:40:22.781324 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:41:06.991097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:06.991070 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp"] Jul 09 16:41:06.994424 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:06.994405 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:06.997433 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:06.997412 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-36b77c4f-openshift-default-dockercfg-tdftz\"" Jul 09 16:41:07.015218 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.015196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp"] Jul 09 16:41:07.080898 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.080852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-envoy\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081025 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.080908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9v4\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-kube-api-access-bz9v4\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081025 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.080967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-certs\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081025 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.081003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-data\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081235 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.081030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-socket\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081235 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.081052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-token\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081235 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.081084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-podinfo\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081235 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.081183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-credential-socket\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.081235 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.081220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ee31a9-481a-418b-854d-9d921fa39af9-istiod-ca-cert\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-credential-socket\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ee31a9-481a-418b-854d-9d921fa39af9-istiod-ca-cert\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-envoy\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9v4\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-kube-api-access-bz9v4\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-certs\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-data\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-socket\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-token\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182515 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-podinfo\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182855 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-credential-socket\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.182855 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182820 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-data\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.183002 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.182979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-certs\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.183083 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.183053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-socket\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.183258 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.183241 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ee31a9-481a-418b-854d-9d921fa39af9-istiod-ca-cert\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.184504 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.184488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-envoy\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.184756 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.184741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-podinfo\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.198594 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.198566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9v4\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-kube-api-access-bz9v4\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.198685 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.198571 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-token\") pod \"e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.303611 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.303558 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:07.433057 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.433035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp"] Jul 09 16:41:07.435511 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:41:07.435480 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ee31a9_481a_418b_854d_9d921fa39af9.slice/crio-bb9147aef3a9be7a345118824c1c876f4ffdccd92b152a5e8b3fe335404c1ef8 WatchSource:0}: Error finding container bb9147aef3a9be7a345118824c1c876f4ffdccd92b152a5e8b3fe335404c1ef8: Status 404 returned error can't find the container with id bb9147aef3a9be7a345118824c1c876f4ffdccd92b152a5e8b3fe335404c1ef8 Jul 09 16:41:07.437560 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.437531 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:07.437623 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.437595 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:07.437659 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.437629 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:07.661469 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.661430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" event={"ID":"c6ee31a9-481a-418b-854d-9d921fa39af9","Type":"ContainerStarted","Data":"81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed"} Jul 09 16:41:07.661660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.661503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" event={"ID":"c6ee31a9-481a-418b-854d-9d921fa39af9","Type":"ContainerStarted","Data":"bb9147aef3a9be7a345118824c1c876f4ffdccd92b152a5e8b3fe335404c1ef8"} Jul 09 16:41:07.683489 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:07.683440 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" podStartSLOduration=1.683425789 podStartE2EDuration="1.683425789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:07.681271955 +0000 UTC m=+1545.479492510" watchObservedRunningTime="2026-07-09 16:41:07.683425789 +0000 UTC m=+1545.481646330" Jul 09 16:41:08.303944 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:08.303913 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:08.308547 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:08.308525 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:08.666666 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:08.666640 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:08.667558 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:08.667541 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:13.433582 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:13.433536 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp"] Jul 09 16:41:13.434171 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:13.433767 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" podUID="c6ee31a9-481a-418b-854d-9d921fa39af9" containerName="istio-proxy" containerID="cri-o://81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed" gracePeriod=30 Jul 09 16:41:19.183294 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.183266 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:19.296180 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296125 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-envoy\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296180 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-certs\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296214 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ee31a9-481a-418b-854d-9d921fa39af9-istiod-ca-cert\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296244 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-socket\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296271 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz9v4\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-kube-api-access-bz9v4\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296299 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-podinfo\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296521 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296442 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-credential-socket\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296521 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296485 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:19.296625 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296525 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-token\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.296822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296582 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:19.296822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296677 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ee31a9-481a-418b-854d-9d921fa39af9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:41:19.296987 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:19.296987 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.296798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-data\") pod \"c6ee31a9-481a-418b-854d-9d921fa39af9\" (UID: \"c6ee31a9-481a-418b-854d-9d921fa39af9\") " Jul 09 16:41:19.297226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.297205 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.297308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.297231 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ee31a9-481a-418b-854d-9d921fa39af9-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.297308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.297244 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.297308 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.297257 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.297480 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.297457 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:19.298438 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.298412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:41:19.298529 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.298500 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-kube-api-access-bz9v4" (OuterVolumeSpecName: "kube-api-access-bz9v4") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "kube-api-access-bz9v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:19.298588 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.298524 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:19.298674 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.298660 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6ee31a9-481a-418b-854d-9d921fa39af9" (UID: "c6ee31a9-481a-418b-854d-9d921fa39af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:19.398624 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.398603 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.398624 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.398623 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.398752 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.398632 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz9v4\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-kube-api-access-bz9v4\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.398752 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.398642 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.398752 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.398649 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ee31a9-481a-418b-854d-9d921fa39af9-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:19.711544 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.711515 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx"] Jul 09 16:41:19.711904 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.711873 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ee31a9-481a-418b-854d-9d921fa39af9" containerName="istio-proxy" Jul 09 16:41:19.711904 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.711903 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ee31a9-481a-418b-854d-9d921fa39af9" containerName="istio-proxy" Jul 09 16:41:19.712039 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.711957 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ee31a9-481a-418b-854d-9d921fa39af9" containerName="istio-proxy" Jul 09 16:41:19.715024 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.715004 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.718715 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.718695 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e2c0cec8-openshift-default-dockercfg-kxr4l\"" Jul 09 16:41:19.721053 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.721027 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6ee31a9-481a-418b-854d-9d921fa39af9" containerID="81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed" exitCode=0 Jul 09 16:41:19.721142 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.721071 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" Jul 09 16:41:19.721142 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.721114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" event={"ID":"c6ee31a9-481a-418b-854d-9d921fa39af9","Type":"ContainerDied","Data":"81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed"} Jul 09 16:41:19.721260 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.721169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp" event={"ID":"c6ee31a9-481a-418b-854d-9d921fa39af9","Type":"ContainerDied","Data":"bb9147aef3a9be7a345118824c1c876f4ffdccd92b152a5e8b3fe335404c1ef8"} Jul 09 16:41:19.721260 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.721204 2562 scope.go:117] "RemoveContainer" containerID="81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed" Jul 09 16:41:19.737725 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.737700 2562 scope.go:117] "RemoveContainer" containerID="81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed" Jul 09 16:41:19.738083 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:41:19.738058 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed\": container with ID starting with 81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed not found: ID does not exist" containerID="81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed" Jul 09 16:41:19.738193 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.738093 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed"} err="failed to get container status \"81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed\": rpc error: code = NotFound desc = could not find container \"81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed\": container with ID starting with 81f3e72ea0980fb6a700370cd2ff5898b1f54a4cda0c79d1588716e955ae57ed not found: ID does not exist" Jul 09 16:41:19.738316 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.738298 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx"] Jul 09 16:41:19.827914 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.827832 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp"] Jul 09 16:41:19.832785 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.832739 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-36b77c4f-openshift-default-dfbcd86d7-4c4tp"] Jul 09 16:41:19.902214 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-certs\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902214 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-podinfo\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902405 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-credential-socket\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902405 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-token\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902405 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vxw\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-kube-api-access-d6vxw\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902547 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cd12d3-e029-4e76-85d8-36d47437bec1-istiod-ca-cert\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902547 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-socket\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902547 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-envoy\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:19.902667 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:19.902547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-data\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.003746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.003679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-envoy\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.003746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.003712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-data\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.003959 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.003825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-certs\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.003959 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.003859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-podinfo\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004064 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.003983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-credential-socket\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004064 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-token\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004064 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vxw\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-kube-api-access-d6vxw\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004203 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cd12d3-e029-4e76-85d8-36d47437bec1-istiod-ca-cert\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004203 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-socket\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004203 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004180 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-data\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-certs\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004600 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-credential-socket\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004708 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-socket\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.004782 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.004766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cd12d3-e029-4e76-85d8-36d47437bec1-istiod-ca-cert\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.005928 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.005908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-envoy\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.006330 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.006309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-podinfo\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.015471 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.015445 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-token\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.015562 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.015501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vxw\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-kube-api-access-d6vxw\") pod \"e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.029072 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.029049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:20.157951 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.157925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx"] Jul 09 16:41:20.159359 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:41:20.159335 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cd12d3_e029_4e76_85d8_36d47437bec1.slice/crio-957471a93456dc90e3e2cf99e61ef8577fd521d0c131b99a0cf13d1e544cd7c3 WatchSource:0}: Error finding container 957471a93456dc90e3e2cf99e61ef8577fd521d0c131b99a0cf13d1e544cd7c3: Status 404 returned error can't find the container with id 957471a93456dc90e3e2cf99e61ef8577fd521d0c131b99a0cf13d1e544cd7c3 Jul 09 16:41:20.161436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.161407 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:20.161508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.161469 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:20.161508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.161502 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:20.727792 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.727750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" event={"ID":"e7cd12d3-e029-4e76-85d8-36d47437bec1","Type":"ContainerStarted","Data":"2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172"} Jul 09 16:41:20.728252 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.727813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" event={"ID":"e7cd12d3-e029-4e76-85d8-36d47437bec1","Type":"ContainerStarted","Data":"957471a93456dc90e3e2cf99e61ef8577fd521d0c131b99a0cf13d1e544cd7c3"} Jul 09 16:41:20.753262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.753219 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" podStartSLOduration=1.753205178 podStartE2EDuration="1.753205178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:20.7504209 +0000 UTC m=+1558.548641465" watchObservedRunningTime="2026-07-09 16:41:20.753205178 +0000 UTC m=+1558.551425719" Jul 09 16:41:20.820511 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:20.820487 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ee31a9-481a-418b-854d-9d921fa39af9" path="/var/lib/kubelet/pods/c6ee31a9-481a-418b-854d-9d921fa39af9/volumes" Jul 09 16:41:21.029692 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:21.029600 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:21.034944 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:21.034921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:21.733361 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:21.733327 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:21.734405 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:21.734382 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:32.076498 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.076388 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx"] Jul 09 16:41:32.076932 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.076587 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" podUID="e7cd12d3-e029-4e76-85d8-36d47437bec1" containerName="istio-proxy" containerID="cri-o://2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172" gracePeriod=30 Jul 09 16:41:32.229822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.229793 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:32.317756 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cd12d3-e029-4e76-85d8-36d47437bec1-istiod-ca-cert\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.317941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317775 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-certs\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.317941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-envoy\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.317941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317840 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-credential-socket\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.317941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317861 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6vxw\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-kube-api-access-d6vxw\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.317941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317909 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-data\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.317941 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317940 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-podinfo\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.318237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.317982 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-token\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.318237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318017 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-socket\") pod \"e7cd12d3-e029-4e76-85d8-36d47437bec1\" (UID: \"e7cd12d3-e029-4e76-85d8-36d47437bec1\") " Jul 09 16:41:32.318237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318084 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cd12d3-e029-4e76-85d8-36d47437bec1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:41:32.318237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318159 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:32.318416 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:32.319082 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318372 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7cd12d3-e029-4e76-85d8-36d47437bec1-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.319082 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318959 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.319082 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.318955 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:32.319082 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.319014 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:32.326259 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.320980 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:32.326259 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.322996 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:41:32.326489 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.326459 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:32.326809 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.326748 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-kube-api-access-d6vxw" (OuterVolumeSpecName: "kube-api-access-d6vxw") pod "e7cd12d3-e029-4e76-85d8-36d47437bec1" (UID: "e7cd12d3-e029-4e76-85d8-36d47437bec1"). InnerVolumeSpecName "kube-api-access-d6vxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:32.420029 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.419994 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.420029 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.420025 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.420029 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.420034 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.420253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.420043 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.420253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.420053 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vxw\" (UniqueName: \"kubernetes.io/projected/e7cd12d3-e029-4e76-85d8-36d47437bec1-kube-api-access-d6vxw\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.420253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.420061 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.420253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.420069 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7cd12d3-e029-4e76-85d8-36d47437bec1-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:32.788987 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.788951 2562 generic.go:356] "Generic (PLEG): container finished" podID="e7cd12d3-e029-4e76-85d8-36d47437bec1" containerID="2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172" exitCode=0 Jul 09 16:41:32.789158 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.789023 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" Jul 09 16:41:32.789158 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.789031 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" event={"ID":"e7cd12d3-e029-4e76-85d8-36d47437bec1","Type":"ContainerDied","Data":"2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172"} Jul 09 16:41:32.789158 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.789120 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx" event={"ID":"e7cd12d3-e029-4e76-85d8-36d47437bec1","Type":"ContainerDied","Data":"957471a93456dc90e3e2cf99e61ef8577fd521d0c131b99a0cf13d1e544cd7c3"} Jul 09 16:41:32.789158 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.789147 2562 scope.go:117] "RemoveContainer" containerID="2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172" Jul 09 16:41:32.798624 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.798597 2562 scope.go:117] "RemoveContainer" containerID="2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172" Jul 09 16:41:32.798936 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:41:32.798916 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172\": container with ID starting with 2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172 not found: ID does not exist" containerID="2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172" Jul 09 16:41:32.799002 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.798945 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172"} err="failed to get container status \"2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172\": rpc error: code = NotFound desc = could not find container \"2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172\": container with ID starting with 2ea1207c169a085e3b063e23f76c9c69a7c1201f057607c66516ca75f332e172 not found: ID does not exist" Jul 09 16:41:32.811754 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.811726 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx"] Jul 09 16:41:32.820606 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:32.820582 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e2c0cec8-openshift-default-59d766fd9-5kwmx"] Jul 09 16:41:34.819457 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:34.819428 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7cd12d3-e029-4e76-85d8-36d47437bec1" path="/var/lib/kubelet/pods/e7cd12d3-e029-4e76-85d8-36d47437bec1/volumes" Jul 09 16:41:38.595451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.595006 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h"] Jul 09 16:41:38.596443 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.596410 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7cd12d3-e029-4e76-85d8-36d47437bec1" containerName="istio-proxy" Jul 09 16:41:38.596443 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.596442 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7cd12d3-e029-4e76-85d8-36d47437bec1" containerName="istio-proxy" Jul 09 16:41:38.596630 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.596551 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7cd12d3-e029-4e76-85d8-36d47437bec1" containerName="istio-proxy" Jul 09 16:41:38.600572 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.600539 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.607076 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.605385 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-62c5cd5e-openshift-default-dockercfg-5dffq\"" Jul 09 16:41:38.614429 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.614395 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h"] Jul 09 16:41:38.680848 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.680803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-credential-socket\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.680865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-socket\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.680965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-certs\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.680987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istiod-ca-cert\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.681031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-data\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.681070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-token\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.681103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwrg\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-kube-api-access-bwwrg\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.681140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-envoy\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.681436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.681195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-podinfo\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-socket\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-certs\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istiod-ca-cert\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-data\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782320 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-token\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwrg\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-kube-api-access-bwwrg\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782376 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-envoy\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782409 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-podinfo\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-credential-socket\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782651 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-socket\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-data\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.782957 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-credential-socket\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.783113 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.782960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-certs\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.783113 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.783064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istiod-ca-cert\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.784623 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.784600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-envoy\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.784786 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.784767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-podinfo\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.791815 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.791789 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-token\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.791968 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.791955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwrg\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-kube-api-access-bwwrg\") pod \"e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:38.915839 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:38.915807 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:39.065998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.065973 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h"] Jul 09 16:41:39.068701 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:41:39.068673 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2181818b_21dd_4521_84fd_c44eeb4ff4a2.slice/crio-eaad3c21a12d6c604e5362254d442d874da6563cfb239e3a217a7c7dad422ebf WatchSource:0}: Error finding container eaad3c21a12d6c604e5362254d442d874da6563cfb239e3a217a7c7dad422ebf: Status 404 returned error can't find the container with id eaad3c21a12d6c604e5362254d442d874da6563cfb239e3a217a7c7dad422ebf Jul 09 16:41:39.071044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.071009 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:39.071105 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.071080 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:39.071137 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.071108 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:41:39.824301 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.824223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" event={"ID":"2181818b-21dd-4521-84fd-c44eeb4ff4a2","Type":"ContainerStarted","Data":"6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4"} Jul 09 16:41:39.824301 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.824287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" event={"ID":"2181818b-21dd-4521-84fd-c44eeb4ff4a2","Type":"ContainerStarted","Data":"eaad3c21a12d6c604e5362254d442d874da6563cfb239e3a217a7c7dad422ebf"} Jul 09 16:41:39.916902 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:39.916852 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:40.921804 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:40.921777 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:40.944411 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:40.944355 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" podStartSLOduration=2.944334923 podStartE2EDuration="2.944334923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:39.85397618 +0000 UTC m=+1577.652196725" watchObservedRunningTime="2026-07-09 16:41:40.944334923 +0000 UTC m=+1578.742555464" Jul 09 16:41:41.835246 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:41.835211 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:41.836276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:41.836254 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:44.358975 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:44.358938 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h"] Jul 09 16:41:44.854746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:44.854549 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" podUID="2181818b-21dd-4521-84fd-c44eeb4ff4a2" containerName="istio-proxy" containerID="cri-o://6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4" gracePeriod=30 Jul 09 16:41:50.106659 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.106634 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:50.192175 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192143 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-envoy\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192175 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192181 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwwrg\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-kube-api-access-bwwrg\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-credential-socket\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192231 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-data\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-token\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-socket\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192340 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-certs\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192625 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192392 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istiod-ca-cert\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192625 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192416 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-podinfo\") pod \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\" (UID: \"2181818b-21dd-4521-84fd-c44eeb4ff4a2\") " Jul 09 16:41:50.192625 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:50.192625 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192544 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:50.192822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192776 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:50.192822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:41:50.192822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192800 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.192822 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192818 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.193039 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.192840 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:50.194413 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.194384 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:41:50.194413 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.194398 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-kube-api-access-bwwrg" (OuterVolumeSpecName: "kube-api-access-bwwrg") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "kube-api-access-bwwrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:50.194692 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.194663 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:41:50.194692 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.194672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "2181818b-21dd-4521-84fd-c44eeb4ff4a2" (UID: "2181818b-21dd-4521-84fd-c44eeb4ff4a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293766 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293803 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293820 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293841 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293857 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwwrg\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-kube-api-access-bwwrg\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293871 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181818b-21dd-4521-84fd-c44eeb4ff4a2-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.296397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.293913 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181818b-21dd-4521-84fd-c44eeb4ff4a2-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:41:50.885274 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.885241 2562 generic.go:356] "Generic (PLEG): container finished" podID="2181818b-21dd-4521-84fd-c44eeb4ff4a2" containerID="6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4" exitCode=0 Jul 09 16:41:50.885457 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.885306 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" Jul 09 16:41:50.885457 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.885337 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" event={"ID":"2181818b-21dd-4521-84fd-c44eeb4ff4a2","Type":"ContainerDied","Data":"6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4"} Jul 09 16:41:50.885457 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.885404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h" event={"ID":"2181818b-21dd-4521-84fd-c44eeb4ff4a2","Type":"ContainerDied","Data":"eaad3c21a12d6c604e5362254d442d874da6563cfb239e3a217a7c7dad422ebf"} Jul 09 16:41:50.885457 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.885440 2562 scope.go:117] "RemoveContainer" containerID="6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4" Jul 09 16:41:50.893991 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.893971 2562 scope.go:117] "RemoveContainer" containerID="6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4" Jul 09 16:41:50.894270 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:41:50.894252 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4\": container with ID starting with 6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4 not found: ID does not exist" containerID="6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4" Jul 09 16:41:50.894326 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.894286 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4"} err="failed to get container status \"6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4\": rpc error: code = NotFound desc = could not find container \"6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4\": container with ID starting with 6176a26d2bab6b086df4286c9e97a3d86de7655f93581bd19e50b376095cecf4 not found: ID does not exist" Jul 09 16:41:50.907634 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.907597 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h"] Jul 09 16:41:50.913291 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:50.913263 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-62c5cd5e-openshift-default-57d8b4bbf9-rs56h"] Jul 09 16:41:52.819543 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:41:52.819511 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2181818b-21dd-4521-84fd-c44eeb4ff4a2" path="/var/lib/kubelet/pods/2181818b-21dd-4521-84fd-c44eeb4ff4a2/volumes" Jul 09 16:43:26.109247 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.109216 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk"] Jul 09 16:43:26.109722 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.109572 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2181818b-21dd-4521-84fd-c44eeb4ff4a2" containerName="istio-proxy" Jul 09 16:43:26.109722 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.109587 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2181818b-21dd-4521-84fd-c44eeb4ff4a2" containerName="istio-proxy" Jul 09 16:43:26.109722 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.109663 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2181818b-21dd-4521-84fd-c44eeb4ff4a2" containerName="istio-proxy" Jul 09 16:43:26.112200 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.112184 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.115330 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.115311 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-840481d3-openshift-default-dockercfg-c6kc8\"" Jul 09 16:43:26.140145 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.140114 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk"] Jul 09 16:43:26.269316 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-certs\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269316 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-data\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269535 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cc0f95-2824-4193-9844-d50960034654-istio-podinfo\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269535 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-envoy\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269535 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cc0f95-2824-4193-9844-d50960034654-istiod-ca-cert\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269535 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gw2h\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-kube-api-access-6gw2h\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269535 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-istio-token\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269690 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-credential-socket\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.269690 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.269633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-socket\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-istio-token\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-credential-socket\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370642 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-socket\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370642 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-certs\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370642 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-data\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370642 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cc0f95-2824-4193-9844-d50960034654-istio-podinfo\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370642 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-envoy\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370873 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cc0f95-2824-4193-9844-d50960034654-istiod-ca-cert\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370873 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gw2h\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-kube-api-access-6gw2h\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370873 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-credential-socket\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.370873 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.370841 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-socket\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.371426 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.371398 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-certs\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.371540 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.371436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cc0f95-2824-4193-9844-d50960034654-istiod-ca-cert\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.371540 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.371455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-data\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.373337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.373318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-envoy\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.373439 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.373374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cc0f95-2824-4193-9844-d50960034654-istio-podinfo\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.383905 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.381826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gw2h\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-kube-api-access-6gw2h\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.385723 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.385589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-istio-token\") pod \"e2e-mt-840481d3-openshift-default-75f645858c-cv9hk\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.421789 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.421753 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:26.561200 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.561173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk"] Jul 09 16:43:26.563010 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:43:26.562981 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cc0f95_2824_4193_9844_d50960034654.slice/crio-cb468c47c83e1938fdbd72f3b4f6b5bed657dd5b3367be79143eda8d9b9cf50f WatchSource:0}: Error finding container cb468c47c83e1938fdbd72f3b4f6b5bed657dd5b3367be79143eda8d9b9cf50f: Status 404 returned error can't find the container with id cb468c47c83e1938fdbd72f3b4f6b5bed657dd5b3367be79143eda8d9b9cf50f Jul 09 16:43:26.565161 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.565124 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:43:26.565254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.565192 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:43:26.565254 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:26.565234 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:43:27.325554 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:27.325518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" event={"ID":"04cc0f95-2824-4193-9844-d50960034654","Type":"ContainerStarted","Data":"a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d"} Jul 09 16:43:27.326085 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:27.325579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" event={"ID":"04cc0f95-2824-4193-9844-d50960034654","Type":"ContainerStarted","Data":"cb468c47c83e1938fdbd72f3b4f6b5bed657dd5b3367be79143eda8d9b9cf50f"} Jul 09 16:43:27.368180 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:27.368129 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" podStartSLOduration=1.3681124599999999 podStartE2EDuration="1.36811246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:43:27.36612916 +0000 UTC m=+1685.164349702" watchObservedRunningTime="2026-07-09 16:43:27.36811246 +0000 UTC m=+1685.166333001" Jul 09 16:43:27.422524 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:27.422484 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:27.427764 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:27.427731 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:28.332506 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:28.332471 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:28.333440 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:28.333423 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:42.404213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:42.404178 2562 generic.go:356] "Generic (PLEG): container finished" podID="04cc0f95-2824-4193-9844-d50960034654" containerID="a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d" exitCode=0 Jul 09 16:43:42.404648 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:42.404257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" event={"ID":"04cc0f95-2824-4193-9844-d50960034654","Type":"ContainerDied","Data":"a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d"} Jul 09 16:43:42.404690 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:42.404672 2562 scope.go:117] "RemoveContainer" containerID="a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d" Jul 09 16:43:42.405752 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:42.405720 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:43:42.405872 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:42.405792 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:43:42.405872 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:42.405830 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:43:43.332501 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:43.332459 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:43.411326 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:43.411285 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" event={"ID":"04cc0f95-2824-4193-9844-d50960034654","Type":"ContainerStarted","Data":"dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38"} Jul 09 16:43:43.422703 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:43.422678 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:43.423859 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:43.423834 2562 patch_prober.go:28] interesting pod/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 09 16:43:43.423977 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:43.423894 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 09 16:43:44.423076 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:44.422987 2562 patch_prober.go:28] interesting pod/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 09 16:43:44.423076 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:44.423037 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 09 16:43:45.422542 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:45.422512 2562 patch_prober.go:28] interesting pod/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 09 16:43:45.422719 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:45.422565 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 09 16:43:46.422396 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:46.422364 2562 patch_prober.go:28] interesting pod/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 09 16:43:46.422783 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:46.422417 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 09 16:43:47.427538 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:47.427508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:47.428110 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:47.428063 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:47.428847 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:47.428827 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:47.607133 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:47.607094 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk"] Jul 09 16:43:48.434746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:48.434714 2562 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" secret="" err="secret \"e2e-mt-840481d3-openshift-default-dockercfg-c6kc8\" not found" Jul 09 16:43:49.438861 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:49.438818 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" containerID="cri-o://dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38" gracePeriod=30 Jul 09 16:43:54.694133 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.694107 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:54.736839 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.736754 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-certs\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.736839 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.736803 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-socket\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737062 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.736852 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cc0f95-2824-4193-9844-d50960034654-istiod-ca-cert\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737062 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.736893 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gw2h\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-kube-api-access-6gw2h\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737062 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.736926 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-envoy\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737054 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cc0f95-2824-4193-9844-d50960034654-istio-podinfo\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737097 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-istio-token\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737096 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:43:54.737220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737116 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:43:54.737220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737188 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-data\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737464 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737293 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-credential-socket\") pod \"04cc0f95-2824-4193-9844-d50960034654\" (UID: \"04cc0f95-2824-4193-9844-d50960034654\") " Jul 09 16:43:54.737464 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737392 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cc0f95-2824-4193-9844-d50960034654-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:43:54.737652 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737625 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.737736 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737649 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.737736 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.737667 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04cc0f95-2824-4193-9844-d50960034654-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.738305 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.738272 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:43:54.738537 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.738507 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-data" (OuterVolumeSpecName: "istio-data") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:43:54.739292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.739267 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04cc0f95-2824-4193-9844-d50960034654-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:43:54.739292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.739280 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-kube-api-access-6gw2h" (OuterVolumeSpecName: "kube-api-access-6gw2h") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "kube-api-access-6gw2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:43:54.739532 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.739511 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:43:54.739635 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.739590 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-istio-token" (OuterVolumeSpecName: "istio-token") pod "04cc0f95-2824-4193-9844-d50960034654" (UID: "04cc0f95-2824-4193-9844-d50960034654"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:43:54.838840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.838798 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.838840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.838830 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gw2h\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-kube-api-access-6gw2h\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.838840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.838840 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.838840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.838851 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04cc0f95-2824-4193-9844-d50960034654-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.839112 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.838859 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04cc0f95-2824-4193-9844-d50960034654-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:54.839112 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:54.838869 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04cc0f95-2824-4193-9844-d50960034654-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:43:55.467777 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.467744 2562 generic.go:356] "Generic (PLEG): container finished" podID="04cc0f95-2824-4193-9844-d50960034654" containerID="dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38" exitCode=0 Jul 09 16:43:55.467998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.467793 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" Jul 09 16:43:55.467998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.467835 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" event={"ID":"04cc0f95-2824-4193-9844-d50960034654","Type":"ContainerDied","Data":"dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38"} Jul 09 16:43:55.467998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.467916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk" event={"ID":"04cc0f95-2824-4193-9844-d50960034654","Type":"ContainerDied","Data":"cb468c47c83e1938fdbd72f3b4f6b5bed657dd5b3367be79143eda8d9b9cf50f"} Jul 09 16:43:55.467998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.467948 2562 scope.go:117] "RemoveContainer" containerID="dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38" Jul 09 16:43:55.476739 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.476720 2562 scope.go:117] "RemoveContainer" containerID="a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d" Jul 09 16:43:55.484484 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.484466 2562 scope.go:117] "RemoveContainer" containerID="dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38" Jul 09 16:43:55.484750 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:43:55.484730 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38\": container with ID starting with dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38 not found: ID does not exist" containerID="dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38" Jul 09 16:43:55.484847 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.484757 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38"} err="failed to get container status \"dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38\": rpc error: code = NotFound desc = could not find container \"dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38\": container with ID starting with dcb3905a94d87a6a733199a9f444b72a1c0911d0558544bfe24407e4e667fa38 not found: ID does not exist" Jul 09 16:43:55.484847 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.484775 2562 scope.go:117] "RemoveContainer" containerID="a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d" Jul 09 16:43:55.485511 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:43:55.485453 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d\": container with ID starting with a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d not found: ID does not exist" containerID="a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d" Jul 09 16:43:55.485511 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.485498 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d"} err="failed to get container status \"a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d\": rpc error: code = NotFound desc = could not find container \"a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d\": container with ID starting with a9c703a07235844536297612d65d6c17fc330fb79b9b15cab6558d94b75eb70d not found: ID does not exist" Jul 09 16:43:55.487626 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.487603 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk"] Jul 09 16:43:55.493464 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:55.493439 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-840481d3-openshift-default-75f645858c-cv9hk"] Jul 09 16:43:56.821278 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:43:56.821243 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04cc0f95-2824-4193-9844-d50960034654" path="/var/lib/kubelet/pods/04cc0f95-2824-4193-9844-d50960034654/volumes" Jul 09 16:44:07.446899 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.446848 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg"] Jul 09 16:44:07.447436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.447243 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" Jul 09 16:44:07.447436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.447255 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" Jul 09 16:44:07.447436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.447319 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" Jul 09 16:44:07.447436 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.447329 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" Jul 09 16:44:07.452410 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.452387 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.455309 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.455288 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7b08359f-openshift-default-dockercfg-x6qdq\"" Jul 09 16:44:07.466123 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.466096 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg"] Jul 09 16:44:07.557897 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.557860 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-data\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.557919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-envoy\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.557945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-socket\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.557963 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-podinfo\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.558020 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-credential-socket\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.558063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5735e7e-d862-4ad5-872f-74e27d6574bb-istiod-ca-cert\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.558089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-certs\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.558135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mblk\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-kube-api-access-4mblk\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.558275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.558208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-token\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.658825 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.658786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5735e7e-d862-4ad5-872f-74e27d6574bb-istiod-ca-cert\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.658825 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.658825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-certs\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.658843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mblk\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-kube-api-access-4mblk\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.658867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-token\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.658948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-data\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.658985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-envoy\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-socket\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-podinfo\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-credential-socket\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659307 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-certs\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659402 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-socket\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659571 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-data\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659571 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-credential-socket\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.659668 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.659645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5735e7e-d862-4ad5-872f-74e27d6574bb-istiod-ca-cert\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.661253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.661224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-envoy\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.661481 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.661464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-podinfo\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.675083 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.675049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mblk\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-kube-api-access-4mblk\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.676646 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.676624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-token\") pod \"e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.762767 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.762677 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:07.906676 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.906650 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg"] Jul 09 16:44:07.908534 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:44:07.908507 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5735e7e_d862_4ad5_872f_74e27d6574bb.slice/crio-353e9122925bd95be46cb2ec94f5e17610d362423acb242daf390070941456eb WatchSource:0}: Error finding container 353e9122925bd95be46cb2ec94f5e17610d362423acb242daf390070941456eb: Status 404 returned error can't find the container with id 353e9122925bd95be46cb2ec94f5e17610d362423acb242daf390070941456eb Jul 09 16:44:07.910861 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.910827 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:44:07.910983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.910929 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:44:07.910983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:07.910962 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:44:08.531039 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:08.531000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" event={"ID":"a5735e7e-d862-4ad5-872f-74e27d6574bb","Type":"ContainerStarted","Data":"cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0"} Jul 09 16:44:08.531419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:08.531054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" event={"ID":"a5735e7e-d862-4ad5-872f-74e27d6574bb","Type":"ContainerStarted","Data":"353e9122925bd95be46cb2ec94f5e17610d362423acb242daf390070941456eb"} Jul 09 16:44:08.554421 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:08.554367 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podStartSLOduration=1.554351134 podStartE2EDuration="1.554351134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:44:08.553153785 +0000 UTC m=+1726.351374327" watchObservedRunningTime="2026-07-09 16:44:08.554351134 +0000 UTC m=+1726.352571675" Jul 09 16:44:08.763377 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:08.763338 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:08.768571 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:08.768544 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:09.536934 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:09.536897 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:09.537924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:09.537896 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:23.605208 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:23.605172 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerID="cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0" exitCode=0 Jul 09 16:44:23.605666 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:23.605245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" event={"ID":"a5735e7e-d862-4ad5-872f-74e27d6574bb","Type":"ContainerDied","Data":"cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0"} Jul 09 16:44:23.605666 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:23.605654 2562 scope.go:117] "RemoveContainer" containerID="cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0" Jul 09 16:44:23.606644 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:23.606618 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:44:23.606700 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:23.606685 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:44:23.606756 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:23.606721 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:44:24.536958 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:24.536920 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:24.611636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:24.611601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" event={"ID":"a5735e7e-d862-4ad5-872f-74e27d6574bb","Type":"ContainerStarted","Data":"e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79"} Jul 09 16:44:24.763401 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:24.763362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:24.764466 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:24.764440 2562 patch_prober.go:28] interesting pod/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 09 16:44:24.764589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:24.764490 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 09 16:44:25.763269 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:25.763232 2562 patch_prober.go:28] interesting pod/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 09 16:44:25.763659 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:25.763293 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 09 16:44:26.763667 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:26.763627 2562 patch_prober.go:28] interesting pod/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 09 16:44:26.764090 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:26.763683 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 09 16:44:27.763847 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:27.763809 2562 patch_prober.go:28] interesting pod/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 09 16:44:27.764312 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:27.763909 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 09 16:44:28.763837 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:28.763793 2562 patch_prober.go:28] interesting pod/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 09 16:44:28.764032 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:28.763864 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 09 16:44:29.766869 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:29.766837 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:29.767307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:29.767154 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:44:29.767775 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:44:29.767762 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:45:00.136289 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.136253 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm"] Jul 09 16:45:00.136770 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.136603 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" Jul 09 16:45:00.136770 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.136614 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04cc0f95-2824-4193-9844-d50960034654" containerName="istio-proxy" Jul 09 16:45:00.139824 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.139804 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:45:00.143266 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.143235 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 16:45:00.143387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.143333 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 16:45:00.144589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.144574 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f92zq\"" Jul 09 16:45:00.149566 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.149541 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm"] Jul 09 16:45:00.158102 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.158070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbwm\" (UniqueName: \"kubernetes.io/projected/3aace439-b75e-4c21-8216-df525f43d6cb-kube-api-access-6wbwm\") pod \"maas-api-key-cleanup-29726925-t9zgm\" (UID: \"3aace439-b75e-4c21-8216-df525f43d6cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:45:00.258849 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.258811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbwm\" (UniqueName: \"kubernetes.io/projected/3aace439-b75e-4c21-8216-df525f43d6cb-kube-api-access-6wbwm\") pod \"maas-api-key-cleanup-29726925-t9zgm\" (UID: \"3aace439-b75e-4c21-8216-df525f43d6cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:45:00.268422 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.268397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbwm\" (UniqueName: \"kubernetes.io/projected/3aace439-b75e-4c21-8216-df525f43d6cb-kube-api-access-6wbwm\") pod \"maas-api-key-cleanup-29726925-t9zgm\" (UID: \"3aace439-b75e-4c21-8216-df525f43d6cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:45:00.451600 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.451567 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:45:00.593672 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.593644 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm"] Jul 09 16:45:00.595849 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:45:00.595827 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aace439_b75e_4c21_8216_df525f43d6cb.slice/crio-0d82a4c805301636721c91edd6a30241d5cc8bd24d53ef8f097b622f89bb8b4b WatchSource:0}: Error finding container 0d82a4c805301636721c91edd6a30241d5cc8bd24d53ef8f097b622f89bb8b4b: Status 404 returned error can't find the container with id 0d82a4c805301636721c91edd6a30241d5cc8bd24d53ef8f097b622f89bb8b4b Jul 09 16:45:00.598548 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.598528 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:45:00.791519 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:00.791423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" event={"ID":"3aace439-b75e-4c21-8216-df525f43d6cb","Type":"ContainerStarted","Data":"0d82a4c805301636721c91edd6a30241d5cc8bd24d53ef8f097b622f89bb8b4b"} Jul 09 16:45:03.808206 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:03.808113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" event={"ID":"3aace439-b75e-4c21-8216-df525f43d6cb","Type":"ContainerStarted","Data":"2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e"} Jul 09 16:45:03.827817 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:03.827755 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" podStartSLOduration=1.296330513 podStartE2EDuration="3.827735939s" totalImagesPullingTime="2.531405426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:45:03.826525117 +0000 UTC m=+1781.624745660" watchObservedRunningTime="2026-07-09 16:45:03.827735939 +0000 UTC m=+1781.625956481" Jul 09 16:45:22.804833 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:22.804801 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:45:22.808706 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:45:22.808675 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:46:00.012206 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:00.012168 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm"] Jul 09 16:46:00.012725 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:00.012375 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" podUID="3aace439-b75e-4c21-8216-df525f43d6cb" containerName="cleanup" containerID="cri-o://2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e" gracePeriod=30 Jul 09 16:46:02.250233 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:02.250193 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg"] Jul 09 16:46:02.250760 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:02.250519 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" containerID="cri-o://e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79" gracePeriod=30 Jul 09 16:46:07.498414 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.498386 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:46:07.590800 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590712 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-data\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.590800 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mblk\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-kube-api-access-4mblk\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.590800 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590778 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-envoy\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590845 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-credential-socket\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590874 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-socket\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590938 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5735e7e-d862-4ad5-872f-74e27d6574bb-istiod-ca-cert\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.590969 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-certs\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591018 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-podinfo\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591077 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-token\") pod \"a5735e7e-d862-4ad5-872f-74e27d6574bb\" (UID: \"a5735e7e-d862-4ad5-872f-74e27d6574bb\") " Jul 09 16:46:07.591502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591210 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:46:07.591502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591316 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:46:07.591502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591380 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5735e7e-d862-4ad5-872f-74e27d6574bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:46:07.591502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591448 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:46:07.591502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591470 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.591713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591503 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5735e7e-d862-4ad5-872f-74e27d6574bb-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.591713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591521 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.591713 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.591656 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:46:07.593215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.593192 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:46:07.593215 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.593210 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:46:07.593450 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.593427 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-kube-api-access-4mblk" (OuterVolumeSpecName: "kube-api-access-4mblk") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "kube-api-access-4mblk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:46:07.593450 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.593432 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5735e7e-d862-4ad5-872f-74e27d6574bb" (UID: "a5735e7e-d862-4ad5-872f-74e27d6574bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:46:07.692272 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.692217 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.692272 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.692263 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.692272 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.692273 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.692272 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.692281 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mblk\" (UniqueName: \"kubernetes.io/projected/a5735e7e-d862-4ad5-872f-74e27d6574bb-kube-api-access-4mblk\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.692272 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.692290 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:07.692594 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:07.692298 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5735e7e-d862-4ad5-872f-74e27d6574bb-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:08.118987 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.118951 2562 generic.go:356] "Generic (PLEG): container finished" podID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerID="e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79" exitCode=0 Jul 09 16:46:08.119173 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.118999 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" Jul 09 16:46:08.119173 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.119005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" event={"ID":"a5735e7e-d862-4ad5-872f-74e27d6574bb","Type":"ContainerDied","Data":"e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79"} Jul 09 16:46:08.119173 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.119057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg" event={"ID":"a5735e7e-d862-4ad5-872f-74e27d6574bb","Type":"ContainerDied","Data":"353e9122925bd95be46cb2ec94f5e17610d362423acb242daf390070941456eb"} Jul 09 16:46:08.119173 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.119085 2562 scope.go:117] "RemoveContainer" containerID="e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79" Jul 09 16:46:08.128780 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.128761 2562 scope.go:117] "RemoveContainer" containerID="cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0" Jul 09 16:46:08.137801 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.137781 2562 scope.go:117] "RemoveContainer" containerID="e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79" Jul 09 16:46:08.138142 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:46:08.138119 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79\": container with ID starting with e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79 not found: ID does not exist" containerID="e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79" Jul 09 16:46:08.138237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.138149 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79"} err="failed to get container status \"e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79\": rpc error: code = NotFound desc = could not find container \"e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79\": container with ID starting with e71d291632915bc4679f2d38a4d7104c29d1d5a69d15093bd89c7f930cef3b79 not found: ID does not exist" Jul 09 16:46:08.138237 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.138169 2562 scope.go:117] "RemoveContainer" containerID="cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0" Jul 09 16:46:08.138406 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:46:08.138387 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0\": container with ID starting with cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0 not found: ID does not exist" containerID="cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0" Jul 09 16:46:08.138443 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.138411 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0"} err="failed to get container status \"cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0\": rpc error: code = NotFound desc = could not find container \"cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0\": container with ID starting with cc4efee14160f5bb617a2109fa2e857b91f6cdc3e5f9db9631497f86549f5ac0 not found: ID does not exist" Jul 09 16:46:08.142207 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.142180 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg"] Jul 09 16:46:08.148852 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.148822 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7b08359f-openshift-default-77bc97ddc5-wwbkg"] Jul 09 16:46:08.820142 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:08.820101 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" path="/var/lib/kubelet/pods/a5735e7e-d862-4ad5-872f-74e27d6574bb/volumes" Jul 09 16:46:30.194821 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.194795 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:46:30.225027 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.224992 2562 generic.go:356] "Generic (PLEG): container finished" podID="3aace439-b75e-4c21-8216-df525f43d6cb" containerID="2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e" exitCode=137 Jul 09 16:46:30.225027 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.225027 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" Jul 09 16:46:30.225230 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.225069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" event={"ID":"3aace439-b75e-4c21-8216-df525f43d6cb","Type":"ContainerDied","Data":"2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e"} Jul 09 16:46:30.225230 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.225119 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm" event={"ID":"3aace439-b75e-4c21-8216-df525f43d6cb","Type":"ContainerDied","Data":"0d82a4c805301636721c91edd6a30241d5cc8bd24d53ef8f097b622f89bb8b4b"} Jul 09 16:46:30.225230 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.225144 2562 scope.go:117] "RemoveContainer" containerID="2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e" Jul 09 16:46:30.233392 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.233371 2562 scope.go:117] "RemoveContainer" containerID="2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e" Jul 09 16:46:30.233659 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:46:30.233639 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e\": container with ID starting with 2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e not found: ID does not exist" containerID="2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e" Jul 09 16:46:30.233741 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.233667 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e"} err="failed to get container status \"2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e\": rpc error: code = NotFound desc = could not find container \"2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e\": container with ID starting with 2162a72f9d1df92dd7283c396b4dd5b1882d943ac0166c7fc98825ef9d7d466e not found: ID does not exist" Jul 09 16:46:30.294281 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.294193 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wbwm\" (UniqueName: \"kubernetes.io/projected/3aace439-b75e-4c21-8216-df525f43d6cb-kube-api-access-6wbwm\") pod \"3aace439-b75e-4c21-8216-df525f43d6cb\" (UID: \"3aace439-b75e-4c21-8216-df525f43d6cb\") " Jul 09 16:46:30.296392 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.296361 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aace439-b75e-4c21-8216-df525f43d6cb-kube-api-access-6wbwm" (OuterVolumeSpecName: "kube-api-access-6wbwm") pod "3aace439-b75e-4c21-8216-df525f43d6cb" (UID: "3aace439-b75e-4c21-8216-df525f43d6cb"). InnerVolumeSpecName "kube-api-access-6wbwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:46:30.395272 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.395236 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wbwm\" (UniqueName: \"kubernetes.io/projected/3aace439-b75e-4c21-8216-df525f43d6cb-kube-api-access-6wbwm\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:46:30.554278 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.554192 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm"] Jul 09 16:46:30.559041 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.559017 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726925-t9zgm"] Jul 09 16:46:30.820241 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:30.820141 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aace439-b75e-4c21-8216-df525f43d6cb" path="/var/lib/kubelet/pods/3aace439-b75e-4c21-8216-df525f43d6cb/volumes" Jul 09 16:46:57.243577 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243496 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn"] Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243863 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243894 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243919 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aace439-b75e-4c21-8216-df525f43d6cb" containerName="cleanup" Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243924 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aace439-b75e-4c21-8216-df525f43d6cb" containerName="cleanup" Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243983 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243993 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" Jul 09 16:46:57.244381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.243998 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aace439-b75e-4c21-8216-df525f43d6cb" containerName="cleanup" Jul 09 16:46:57.246860 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.246843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.250381 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.250362 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6c0f57-openshift-default-dockercfg-6fb9c\"" Jul 09 16:46:57.270095 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.270064 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn"] Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344186 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-credential-socket\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-socket\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bqj\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-kube-api-access-65bqj\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344335 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-envoy\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-data\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-token\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3de1e65-9f56-47dc-b98a-6b4790e67892-istiod-ca-cert\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-certs\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.344673 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.344571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-podinfo\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.445619 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445575 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-credential-socket\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.445619 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-socket\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.445840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bqj\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-kube-api-access-65bqj\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.445840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-envoy\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.445840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445830 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-data\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446023 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-token\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446023 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3de1e65-9f56-47dc-b98a-6b4790e67892-istiod-ca-cert\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446023 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-certs\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446023 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.445995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-podinfo\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446305 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.446048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-credential-socket\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446305 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.446090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-socket\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446305 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.446269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-data\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446472 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.446342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-certs\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.446679 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.446658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3de1e65-9f56-47dc-b98a-6b4790e67892-istiod-ca-cert\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.448082 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.448054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-envoy\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.448384 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.448364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-podinfo\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.455730 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.455702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-token\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.455914 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.455751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bqj\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-kube-api-access-65bqj\") pod \"e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.557031 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.556941 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:57.701432 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.701405 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn"] Jul 09 16:46:57.703233 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:46:57.703203 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3de1e65_9f56_47dc_b98a_6b4790e67892.slice/crio-e2b44fc6cb58dc9696ceff015f22a01aeced825c3b1099ae64987219b37b77d3 WatchSource:0}: Error finding container e2b44fc6cb58dc9696ceff015f22a01aeced825c3b1099ae64987219b37b77d3: Status 404 returned error can't find the container with id e2b44fc6cb58dc9696ceff015f22a01aeced825c3b1099ae64987219b37b77d3 Jul 09 16:46:57.705364 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.705332 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:46:57.705453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.705397 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:46:57.705453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:57.705426 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:46:58.357111 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:58.357070 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" event={"ID":"e3de1e65-9f56-47dc-b98a-6b4790e67892","Type":"ContainerStarted","Data":"4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a"} Jul 09 16:46:58.357511 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:58.357136 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" event={"ID":"e3de1e65-9f56-47dc-b98a-6b4790e67892","Type":"ContainerStarted","Data":"e2b44fc6cb58dc9696ceff015f22a01aeced825c3b1099ae64987219b37b77d3"} Jul 09 16:46:58.381247 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:58.381190 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podStartSLOduration=1.3811697889999999 podStartE2EDuration="1.381169789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:46:58.379775744 +0000 UTC m=+1896.177996287" watchObservedRunningTime="2026-07-09 16:46:58.381169789 +0000 UTC m=+1896.179390328" Jul 09 16:46:58.557484 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:58.557442 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:58.562187 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:58.562163 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:59.361947 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:59.361919 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:46:59.362983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:46:59.362951 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:47:05.463812 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.463773 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw"] Jul 09 16:47:05.464314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.464179 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" Jul 09 16:47:05.464314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.464193 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5735e7e-d862-4ad5-872f-74e27d6574bb" containerName="istio-proxy" Jul 09 16:47:05.467443 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.467420 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.470791 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.470767 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-aa22b3-openshift-default-dockercfg-sl8v7\"" Jul 09 16:47:05.487948 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.487920 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw"] Jul 09 16:47:05.628130 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-credential-socket\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-token\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-envoy\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istiod-ca-cert\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-socket\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kr8\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-kube-api-access-q7kr8\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628593 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-certs\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628593 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-podinfo\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.628593 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.628449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-data\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729231 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-credential-socket\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729231 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-token\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729231 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-envoy\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729238 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istiod-ca-cert\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-socket\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kr8\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-kube-api-access-q7kr8\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-certs\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729724 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-podinfo\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729724 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-data\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729724 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-credential-socket\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.729945 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-socket\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.730040 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-certs\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.730040 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.729989 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-data\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.730168 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.730145 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istiod-ca-cert\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.731651 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.731632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-envoy\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.732171 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.732151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-podinfo\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.738181 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.738147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-token\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.738556 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.738537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kr8\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-kube-api-access-q7kr8\") pod \"e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.778629 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.778596 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:05.930599 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.930566 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw"] Jul 09 16:47:05.931582 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:47:05.931549 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1266777_3276_4c9a_a0ad_e4a8d3d45765.slice/crio-a0d1618d652139c29855084059f53643773d4f1ad6bbdf95cb86a409385aa08b WatchSource:0}: Error finding container a0d1618d652139c29855084059f53643773d4f1ad6bbdf95cb86a409385aa08b: Status 404 returned error can't find the container with id a0d1618d652139c29855084059f53643773d4f1ad6bbdf95cb86a409385aa08b Jul 09 16:47:05.933789 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.933761 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:05.933898 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.933839 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:05.933969 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:05.933918 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:06.396660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:06.396568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" event={"ID":"f1266777-3276-4c9a-a0ad-e4a8d3d45765","Type":"ContainerStarted","Data":"40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca"} Jul 09 16:47:06.396660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:06.396622 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" event={"ID":"f1266777-3276-4c9a-a0ad-e4a8d3d45765","Type":"ContainerStarted","Data":"a0d1618d652139c29855084059f53643773d4f1ad6bbdf95cb86a409385aa08b"} Jul 09 16:47:06.417304 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:06.417249 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" podStartSLOduration=1.417232354 podStartE2EDuration="1.417232354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:47:06.415795556 +0000 UTC m=+1904.214016099" watchObservedRunningTime="2026-07-09 16:47:06.417232354 +0000 UTC m=+1904.215452896" Jul 09 16:47:06.779077 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:06.779030 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:06.784096 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:06.784069 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:07.405829 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:07.405801 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:07.406991 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:07.406965 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:15.404083 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:15.404038 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 16:47:15.404484 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:15.404120 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 16:47:17.463778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:17.463741 2562 generic.go:356] "Generic (PLEG): container finished" podID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerID="4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a" exitCode=0 Jul 09 16:47:17.464397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:17.463784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" event={"ID":"e3de1e65-9f56-47dc-b98a-6b4790e67892","Type":"ContainerDied","Data":"4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a"} Jul 09 16:47:17.464397 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:17.464305 2562 scope.go:117] "RemoveContainer" containerID="4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a" Jul 09 16:47:17.465466 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:17.465439 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:17.465583 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:17.465517 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:17.465583 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:17.465560 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:18.474549 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:18.474495 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" event={"ID":"e3de1e65-9f56-47dc-b98a-6b4790e67892","Type":"ContainerStarted","Data":"379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007"} Jul 09 16:47:18.557755 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:18.557718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:47:18.558144 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:18.558057 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 09 16:47:18.558360 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:18.558335 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 09 16:47:19.557521 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:19.557475 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 09 16:47:19.558184 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:19.558148 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 09 16:47:20.558276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:20.557791 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 09 16:47:20.558276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:20.557863 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 09 16:47:21.557411 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:21.557369 2562 patch_prober.go:28] interesting pod/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 09 16:47:21.557759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:21.557726 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 09 16:47:22.561337 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:22.561295 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:47:22.561761 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:22.561739 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:47:22.562374 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:22.562354 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:47:25.520264 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:25.520230 2562 generic.go:356] "Generic (PLEG): container finished" podID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerID="40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca" exitCode=0 Jul 09 16:47:25.520264 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:25.520267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" event={"ID":"f1266777-3276-4c9a-a0ad-e4a8d3d45765","Type":"ContainerDied","Data":"40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca"} Jul 09 16:47:25.520811 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:25.520663 2562 scope.go:117] "RemoveContainer" containerID="40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca" Jul 09 16:47:25.521655 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:25.521626 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:25.521715 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:25.521694 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:25.521750 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:25.521733 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:47:26.527263 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:26.527218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" event={"ID":"f1266777-3276-4c9a-a0ad-e4a8d3d45765","Type":"ContainerStarted","Data":"8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043"} Jul 09 16:47:26.779165 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:26.779047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:26.779374 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:26.779352 2562 patch_prober.go:28] interesting pod/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 09 16:47:26.779427 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:26.779404 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 09 16:47:27.779568 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:27.779538 2562 patch_prober.go:28] interesting pod/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 09 16:47:27.779966 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:27.779591 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 09 16:47:28.779804 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:28.779764 2562 patch_prober.go:28] interesting pod/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 09 16:47:28.780233 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:28.779823 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 09 16:47:29.779681 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:29.779647 2562 patch_prober.go:28] interesting pod/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 09 16:47:29.779916 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:29.779706 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 09 16:47:30.783225 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:30.783187 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:30.783687 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:30.783643 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:47:30.784451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:47:30.784431 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:49:00.197262 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:00.197220 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn"] Jul 09 16:49:00.197737 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:00.197503 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" containerID="cri-o://379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007" gracePeriod=30 Jul 09 16:49:05.440092 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.440060 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:49:05.478769 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.478688 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65bqj\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-kube-api-access-65bqj\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.478769 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.478724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-envoy\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.478769 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.478751 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-credential-socket\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.478776 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-data\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.478936 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3de1e65-9f56-47dc-b98a-6b4790e67892-istiod-ca-cert\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.478969 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-podinfo\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479029 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-certs\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479074 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-token\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-socket\") pod \"e3de1e65-9f56-47dc-b98a-6b4790e67892\" (UID: \"e3de1e65-9f56-47dc-b98a-6b4790e67892\") " Jul 09 16:49:05.479276 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479159 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:05.479434 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479387 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:05.479434 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479415 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3de1e65-9f56-47dc-b98a-6b4790e67892-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:05.479532 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479516 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.479599 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479535 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3de1e65-9f56-47dc-b98a-6b4790e67892-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.479599 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479549 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.479685 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479655 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:05.479958 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.479917 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:05.481351 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.481323 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:05.481450 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.481426 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:05.481493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.481443 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-kube-api-access-65bqj" (OuterVolumeSpecName: "kube-api-access-65bqj") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "kube-api-access-65bqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:05.481745 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.481721 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3de1e65-9f56-47dc-b98a-6b4790e67892" (UID: "e3de1e65-9f56-47dc-b98a-6b4790e67892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:49:05.580717 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.580674 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.580717 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.580705 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.580717 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.580714 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.580717 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.580723 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.581084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.580731 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bqj\" (UniqueName: \"kubernetes.io/projected/e3de1e65-9f56-47dc-b98a-6b4790e67892-kube-api-access-65bqj\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:05.581084 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:05.580742 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3de1e65-9f56-47dc-b98a-6b4790e67892-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:06.004047 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.004011 2562 generic.go:356] "Generic (PLEG): container finished" podID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerID="379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007" exitCode=0 Jul 09 16:49:06.004226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.004056 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" Jul 09 16:49:06.004226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.004100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" event={"ID":"e3de1e65-9f56-47dc-b98a-6b4790e67892","Type":"ContainerDied","Data":"379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007"} Jul 09 16:49:06.004226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.004158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn" event={"ID":"e3de1e65-9f56-47dc-b98a-6b4790e67892","Type":"ContainerDied","Data":"e2b44fc6cb58dc9696ceff015f22a01aeced825c3b1099ae64987219b37b77d3"} Jul 09 16:49:06.004226 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.004182 2562 scope.go:117] "RemoveContainer" containerID="379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007" Jul 09 16:49:06.013122 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.013104 2562 scope.go:117] "RemoveContainer" containerID="4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a" Jul 09 16:49:06.020926 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.020904 2562 scope.go:117] "RemoveContainer" containerID="379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007" Jul 09 16:49:06.021219 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:06.021198 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007\": container with ID starting with 379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007 not found: ID does not exist" containerID="379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007" Jul 09 16:49:06.021264 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.021228 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007"} err="failed to get container status \"379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007\": rpc error: code = NotFound desc = could not find container \"379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007\": container with ID starting with 379af93e97f7101a60637e4345ce10d12869bd76a38bc9caeb62854809e40007 not found: ID does not exist" Jul 09 16:49:06.021264 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.021246 2562 scope.go:117] "RemoveContainer" containerID="4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a" Jul 09 16:49:06.021464 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:06.021450 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a\": container with ID starting with 4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a not found: ID does not exist" containerID="4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a" Jul 09 16:49:06.021506 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.021465 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a"} err="failed to get container status \"4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a\": rpc error: code = NotFound desc = could not find container \"4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a\": container with ID starting with 4cb733d4744ca8b8fb9629321936c654a456cd564bbbae5c715c75df06c31a0a not found: ID does not exist" Jul 09 16:49:06.025917 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.025876 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn"] Jul 09 16:49:06.032320 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.032298 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6c0f57-openshift-default-cc5f85cf5-rm7rn"] Jul 09 16:49:06.820399 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:06.820367 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" path="/var/lib/kubelet/pods/e3de1e65-9f56-47dc-b98a-6b4790e67892/volumes" Jul 09 16:49:07.367259 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:07.367228 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw"] Jul 09 16:49:07.367461 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:07.367439 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" containerID="cri-o://8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043" gracePeriod=30 Jul 09 16:49:08.272250 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272218 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5"] Jul 09 16:49:08.272752 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272734 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" Jul 09 16:49:08.272828 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272756 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" Jul 09 16:49:08.272828 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272773 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" Jul 09 16:49:08.272828 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272780 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" Jul 09 16:49:08.272998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272912 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" Jul 09 16:49:08.272998 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.272928 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3de1e65-9f56-47dc-b98a-6b4790e67892" containerName="istio-proxy" Jul 09 16:49:08.277578 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.277555 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.280952 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.280933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6d03d1-openshift-default-dockercfg-hdggl\"" Jul 09 16:49:08.292419 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.292394 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5"] Jul 09 16:49:08.306060 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istiod-ca-cert\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306182 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-socket\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306182 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-podinfo\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306182 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-token\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-certs\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272fb\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-kube-api-access-272fb\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-envoy\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-data\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.306441 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.306325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-credential-socket\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-certs\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272fb\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-kube-api-access-272fb\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-envoy\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-data\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-credential-socket\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istiod-ca-cert\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-socket\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-podinfo\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407580 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-token\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407989 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-certs\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407989 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-data\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.407989 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.407905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-socket\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.408154 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.408128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-credential-socket\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.408199 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.408149 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istiod-ca-cert\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.410022 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.409995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-envoy\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.410249 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.410227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-podinfo\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.422538 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.422511 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-token\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.423840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.423816 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-272fb\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-kube-api-access-272fb\") pod \"e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.588109 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.588012 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:08.776044 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.775965 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5"] Jul 09 16:49:08.781463 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.781423 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:08.781577 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.781496 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:08.781577 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:08.781526 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:09.023295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:09.023260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" event={"ID":"fc8ea14b-a177-426c-ada0-1c1ca09ac480","Type":"ContainerStarted","Data":"4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4"} Jul 09 16:49:09.023472 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:09.023314 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" event={"ID":"fc8ea14b-a177-426c-ada0-1c1ca09ac480","Type":"ContainerStarted","Data":"1ffb50600e51c5790629a22d4b51bde0549e9fbfd214ac853bf1891a972126b3"} Jul 09 16:49:09.049723 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:09.049669 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podStartSLOduration=1.049652835 podStartE2EDuration="1.049652835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:09.047207972 +0000 UTC m=+2026.845428564" watchObservedRunningTime="2026-07-09 16:49:09.049652835 +0000 UTC m=+2026.847873375" Jul 09 16:49:09.589203 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:09.589162 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:09.594135 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:09.594107 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:10.028356 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:10.028323 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:10.029375 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:10.029355 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:12.623129 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.623085 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:49:12.647282 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647170 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-certs\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647282 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647214 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-credential-socket\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647282 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647250 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istiod-ca-cert\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647338 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-token\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647372 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-data\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647410 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-socket\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647440 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7kr8\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-kube-api-access-q7kr8\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647471 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-podinfo\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647502 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:12.647643 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647586 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:12.648097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.647948 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:12.648163 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.648141 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-envoy\") pod \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\" (UID: \"f1266777-3276-4c9a-a0ad-e4a8d3d45765\") " Jul 09 16:49:12.648943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.648527 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.648943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.648551 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.648943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.648567 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.648943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.648595 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:12.649245 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.649080 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:12.651134 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.651098 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:49:12.651557 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.651532 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:12.651636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.651555 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:12.651636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.651559 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-kube-api-access-q7kr8" (OuterVolumeSpecName: "kube-api-access-q7kr8") pod "f1266777-3276-4c9a-a0ad-e4a8d3d45765" (UID: "f1266777-3276-4c9a-a0ad-e4a8d3d45765"). InnerVolumeSpecName "kube-api-access-q7kr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:12.749336 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.749297 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.749336 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.749329 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.749336 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.749339 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7kr8\" (UniqueName: \"kubernetes.io/projected/f1266777-3276-4c9a-a0ad-e4a8d3d45765-kube-api-access-q7kr8\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.749570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.749348 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.749570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.749358 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:12.749570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:12.749366 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1266777-3276-4c9a-a0ad-e4a8d3d45765-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:13.045052 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.045018 2562 generic.go:356] "Generic (PLEG): container finished" podID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerID="8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043" exitCode=0 Jul 09 16:49:13.045239 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.045073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" event={"ID":"f1266777-3276-4c9a-a0ad-e4a8d3d45765","Type":"ContainerDied","Data":"8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043"} Jul 09 16:49:13.045239 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.045115 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" event={"ID":"f1266777-3276-4c9a-a0ad-e4a8d3d45765","Type":"ContainerDied","Data":"a0d1618d652139c29855084059f53643773d4f1ad6bbdf95cb86a409385aa08b"} Jul 09 16:49:13.045239 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.045141 2562 scope.go:117] "RemoveContainer" containerID="8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043" Jul 09 16:49:13.045239 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.045074 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw" Jul 09 16:49:13.054305 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.054286 2562 scope.go:117] "RemoveContainer" containerID="40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca" Jul 09 16:49:13.062265 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.062174 2562 scope.go:117] "RemoveContainer" containerID="8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043" Jul 09 16:49:13.062771 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:13.062744 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043\": container with ID starting with 8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043 not found: ID does not exist" containerID="8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043" Jul 09 16:49:13.062865 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.062778 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043"} err="failed to get container status \"8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043\": rpc error: code = NotFound desc = could not find container \"8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043\": container with ID starting with 8c305b85c82f2658997d18b737105e278459a65a2767ff2ad86fc61aa9ef0043 not found: ID does not exist" Jul 09 16:49:13.062865 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.062803 2562 scope.go:117] "RemoveContainer" containerID="40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca" Jul 09 16:49:13.063181 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:13.063154 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca\": container with ID starting with 40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca not found: ID does not exist" containerID="40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca" Jul 09 16:49:13.063277 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.063188 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca"} err="failed to get container status \"40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca\": rpc error: code = NotFound desc = could not find container \"40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca\": container with ID starting with 40846bb81eda7069b0dc0cac1271a2121aeb8087749698b5bc42f8480fc6c7ca not found: ID does not exist" Jul 09 16:49:13.064741 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.064721 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw"] Jul 09 16:49:13.070528 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:13.070509 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-aa22b3-openshift-default-594b58c757-hxrdw"] Jul 09 16:49:14.820263 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:14.820232 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" path="/var/lib/kubelet/pods/f1266777-3276-4c9a-a0ad-e4a8d3d45765/volumes" Jul 09 16:49:16.263202 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.263166 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2"] Jul 09 16:49:16.263682 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.263652 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" Jul 09 16:49:16.263682 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.263670 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" Jul 09 16:49:16.263821 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.263801 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" Jul 09 16:49:16.263978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.263906 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" Jul 09 16:49:16.268331 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.268310 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.271314 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.271293 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-554cc9-openshift-default-dockercfg-xcxrn\"" Jul 09 16:49:16.279380 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.279358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2"] Jul 09 16:49:16.389256 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-credential-socket\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgd2w\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-kube-api-access-fgd2w\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-certs\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389363 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istiod-ca-cert\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-token\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-socket\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389476 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-podinfo\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-data\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.389759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.389557 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-envoy\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.490876 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.490841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-envoy\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.490938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-credential-socket\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.490969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgd2w\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-kube-api-access-fgd2w\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.490992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-certs\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istiod-ca-cert\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-token\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-socket\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491148 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-podinfo\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491477 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-data\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491477 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-credential-socket\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491570 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-certs\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491677 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491659 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-socket\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491719 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491705 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-data\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.491858 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.491836 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istiod-ca-cert\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.493247 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.493226 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-envoy\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.493619 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.493600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-podinfo\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.499781 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.499755 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-token\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.500236 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.500213 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgd2w\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-kube-api-access-fgd2w\") pod \"e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.580763 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.580666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:16.721490 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.721463 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2"] Jul 09 16:49:16.724094 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:49:16.724061 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2ec65f_1d9b_4868_9c97_e00b7ad9f2e1.slice/crio-176c2d6150c36bcf4f60ec6bf47b80e424f77a2873beb4680c16d61e34dadcf2 WatchSource:0}: Error finding container 176c2d6150c36bcf4f60ec6bf47b80e424f77a2873beb4680c16d61e34dadcf2: Status 404 returned error can't find the container with id 176c2d6150c36bcf4f60ec6bf47b80e424f77a2873beb4680c16d61e34dadcf2 Jul 09 16:49:16.726486 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.726451 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:16.726589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.726536 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:16.726589 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:16.726577 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:17.066474 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:17.066440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" event={"ID":"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1","Type":"ContainerStarted","Data":"2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3"} Jul 09 16:49:17.066662 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:17.066497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" event={"ID":"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1","Type":"ContainerStarted","Data":"176c2d6150c36bcf4f60ec6bf47b80e424f77a2873beb4680c16d61e34dadcf2"} Jul 09 16:49:17.088695 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:17.088641 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" podStartSLOduration=1.088622103 podStartE2EDuration="1.088622103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:17.087601953 +0000 UTC m=+2034.885822528" watchObservedRunningTime="2026-07-09 16:49:17.088622103 +0000 UTC m=+2034.886842645" Jul 09 16:49:17.581467 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:17.581428 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:17.586509 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:17.586486 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:18.071259 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:18.071226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:18.072328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:18.072308 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:26.069071 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:26.069021 2562 patch_prober.go:28] interesting pod/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 16:49:26.069453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:26.069092 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 16:49:28.121867 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:28.121832 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerID="4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4" exitCode=0 Jul 09 16:49:28.122295 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:28.121908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" event={"ID":"fc8ea14b-a177-426c-ada0-1c1ca09ac480","Type":"ContainerDied","Data":"4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4"} Jul 09 16:49:28.122336 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:28.122319 2562 scope.go:117] "RemoveContainer" containerID="4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4" Jul 09 16:49:28.123321 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:28.123294 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:28.123384 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:28.123362 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:28.123423 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:28.123393 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:29.132009 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:29.131954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" event={"ID":"fc8ea14b-a177-426c-ada0-1c1ca09ac480","Type":"ContainerStarted","Data":"bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1"} Jul 09 16:49:29.588650 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:29.588609 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:29.588923 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:29.588900 2562 patch_prober.go:28] interesting pod/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 09 16:49:29.588988 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:29.588949 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 09 16:49:30.588845 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:30.588807 2562 patch_prober.go:28] interesting pod/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 09 16:49:30.589283 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:30.588874 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 09 16:49:31.588485 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:31.588447 2562 patch_prober.go:28] interesting pod/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 09 16:49:31.588666 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:31.588501 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 09 16:49:32.588629 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:32.588583 2562 patch_prober.go:28] interesting pod/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 09 16:49:32.589151 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:32.588658 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 09 16:49:33.592185 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:33.592158 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:33.592605 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:33.592415 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:33.593147 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:33.593129 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:37.216437 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:37.216407 2562 generic.go:356] "Generic (PLEG): container finished" podID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerID="2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3" exitCode=0 Jul 09 16:49:37.216875 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:37.216486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" event={"ID":"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1","Type":"ContainerDied","Data":"2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3"} Jul 09 16:49:37.216946 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:37.216875 2562 scope.go:117] "RemoveContainer" containerID="2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3" Jul 09 16:49:37.217841 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:37.217818 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:37.217924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:37.217908 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:37.217967 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:37.217945 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:38.220137 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:38.220106 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5"] Jul 09 16:49:38.220622 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:38.220325 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" containerID="cri-o://bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1" gracePeriod=30 Jul 09 16:49:38.224268 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:38.224238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" event={"ID":"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1","Type":"ContainerStarted","Data":"f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608"} Jul 09 16:49:38.581186 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:38.581099 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:38.581342 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:38.581292 2562 patch_prober.go:28] interesting pod/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 09 16:49:38.581390 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:38.581350 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 09 16:49:39.581579 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:39.581539 2562 patch_prober.go:28] interesting pod/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 09 16:49:39.582013 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:39.581596 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 09 16:49:40.582097 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:40.582053 2562 patch_prober.go:28] interesting pod/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 09 16:49:40.582574 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:40.582120 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 09 16:49:41.582149 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:41.582103 2562 patch_prober.go:28] interesting pod/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 09 16:49:41.582586 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:41.582180 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 09 16:49:42.584801 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:42.584768 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:42.585235 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:42.585194 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:42.585983 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:42.585963 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:43.482615 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.482555 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:43.586783 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.586745 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-socket\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.586804 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-272fb\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-kube-api-access-272fb\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.586842 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-certs\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.586870 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-envoy\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.586914 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-credential-socket\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587010 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-token\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587107 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587126 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-podinfo\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587158 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-data\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587201 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istiod-ca-cert\") pod \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\" (UID: \"fc8ea14b-a177-426c-ada0-1c1ca09ac480\") " Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:43.587275 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587270 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:43.587830 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587618 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.587830 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587641 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.587830 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587656 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.587830 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.587669 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:43.588328 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.588305 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:43.589218 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.589187 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:43.589348 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.589294 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:49:43.589542 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.589522 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:43.589595 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.589546 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-kube-api-access-272fb" (OuterVolumeSpecName: "kube-api-access-272fb") pod "fc8ea14b-a177-426c-ada0-1c1ca09ac480" (UID: "fc8ea14b-a177-426c-ada0-1c1ca09ac480"). InnerVolumeSpecName "kube-api-access-272fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:43.688473 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.688437 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.688473 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.688467 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.688473 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.688476 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.688473 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.688485 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.688739 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.688493 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-272fb\" (UniqueName: \"kubernetes.io/projected/fc8ea14b-a177-426c-ada0-1c1ca09ac480-kube-api-access-272fb\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:43.688739 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:43.688502 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8ea14b-a177-426c-ada0-1c1ca09ac480-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:44.260426 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.260339 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerID="bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1" exitCode=0 Jul 09 16:49:44.260426 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.260415 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" Jul 09 16:49:44.260640 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.260436 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" event={"ID":"fc8ea14b-a177-426c-ada0-1c1ca09ac480","Type":"ContainerDied","Data":"bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1"} Jul 09 16:49:44.260640 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.260505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5" event={"ID":"fc8ea14b-a177-426c-ada0-1c1ca09ac480","Type":"ContainerDied","Data":"1ffb50600e51c5790629a22d4b51bde0549e9fbfd214ac853bf1891a972126b3"} Jul 09 16:49:44.260640 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.260538 2562 scope.go:117] "RemoveContainer" containerID="bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1" Jul 09 16:49:44.269664 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.269645 2562 scope.go:117] "RemoveContainer" containerID="4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4" Jul 09 16:49:44.277767 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.277747 2562 scope.go:117] "RemoveContainer" containerID="bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1" Jul 09 16:49:44.278047 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:44.278023 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1\": container with ID starting with bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1 not found: ID does not exist" containerID="bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1" Jul 09 16:49:44.278123 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.278054 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1"} err="failed to get container status \"bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1\": rpc error: code = NotFound desc = could not find container \"bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1\": container with ID starting with bd55460ef18feb8ac59e6da4cc0f9c8d1cfa833d4b92eb85cc44fe6aa93d75d1 not found: ID does not exist" Jul 09 16:49:44.278123 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.278071 2562 scope.go:117] "RemoveContainer" containerID="4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4" Jul 09 16:49:44.278313 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:44.278296 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4\": container with ID starting with 4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4 not found: ID does not exist" containerID="4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4" Jul 09 16:49:44.278354 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.278317 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4"} err="failed to get container status \"4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4\": rpc error: code = NotFound desc = could not find container \"4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4\": container with ID starting with 4a882ffda5263a3625d0ba412b76797cad8d2e85c542bb48016d776a305d38c4 not found: ID does not exist" Jul 09 16:49:44.282708 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.282687 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5"] Jul 09 16:49:44.288113 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.288093 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6d03d1-openshift-default-54c4584955-8vtl5"] Jul 09 16:49:44.820265 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:44.820235 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" path="/var/lib/kubelet/pods/fc8ea14b-a177-426c-ada0-1c1ca09ac480/volumes" Jul 09 16:49:45.082230 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.082125 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2"] Jul 09 16:49:45.266840 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.266801 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" containerID="cri-o://f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608" gracePeriod=30 Jul 09 16:49:45.990508 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990478 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs"] Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990909 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990926 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990937 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990945 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990958 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.990967 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1266777-3276-4c9a-a0ad-e4a8d3d45765" containerName="istio-proxy" Jul 09 16:49:45.991036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.991047 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" Jul 09 16:49:45.991447 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.991058 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8ea14b-a177-426c-ada0-1c1ca09ac480" containerName="istio-proxy" Jul 09 16:49:45.994118 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.994093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:45.997136 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:45.997113 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-bef56b-openshift-default-dockercfg-rblcz\"" Jul 09 16:49:46.006438 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.006401 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs"] Jul 09 16:49:46.111525 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-istio-token\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.111712 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-certs\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.111794 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d6f697-1343-44d9-8912-c55786b4316b-istiod-ca-cert\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.111794 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-envoy\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.111924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-data\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.111924 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-socket\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.112043 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d6f697-1343-44d9-8912-c55786b4316b-istio-podinfo\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.112043 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.111978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-credential-socket\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.112043 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.112016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9rj\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-kube-api-access-fs9rj\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.212799 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.212755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-socket\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.212799 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.212805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d6f697-1343-44d9-8912-c55786b4316b-istio-podinfo\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213094 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.212833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-credential-socket\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213094 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.212862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9rj\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-kube-api-access-fs9rj\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213094 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.212913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-istio-token\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213094 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-certs\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d6f697-1343-44d9-8912-c55786b4316b-istiod-ca-cert\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-envoy\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-data\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-socket\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213292 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-credential-socket\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213543 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213310 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-certs\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213543 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-data\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.213908 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.213857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d6f697-1343-44d9-8912-c55786b4316b-istiod-ca-cert\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.215380 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.215354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-envoy\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.215470 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.215407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d6f697-1343-44d9-8912-c55786b4316b-istio-podinfo\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.221361 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.221334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-istio-token\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.221451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.221436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9rj\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-kube-api-access-fs9rj\") pod \"e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.311330 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.311241 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:46.443390 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.443354 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs"] Jul 09 16:49:46.446175 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:49:46.446147 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d6f697_1343_44d9_8912_c55786b4316b.slice/crio-45ccfbde88904aa056d4f895e090e5fc6d310ac06d9c86c08fbfdcd77ee0f280 WatchSource:0}: Error finding container 45ccfbde88904aa056d4f895e090e5fc6d310ac06d9c86c08fbfdcd77ee0f280: Status 404 returned error can't find the container with id 45ccfbde88904aa056d4f895e090e5fc6d310ac06d9c86c08fbfdcd77ee0f280 Jul 09 16:49:46.448410 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.448370 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:46.448501 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.448467 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:46.448542 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:46.448511 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:47.277729 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:47.277697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" event={"ID":"35d6f697-1343-44d9-8912-c55786b4316b","Type":"ContainerStarted","Data":"35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607"} Jul 09 16:49:47.278149 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:47.277759 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" event={"ID":"35d6f697-1343-44d9-8912-c55786b4316b","Type":"ContainerStarted","Data":"45ccfbde88904aa056d4f895e090e5fc6d310ac06d9c86c08fbfdcd77ee0f280"} Jul 09 16:49:47.304578 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:47.304526 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" podStartSLOduration=2.304508631 podStartE2EDuration="2.304508631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:47.302529093 +0000 UTC m=+2065.100749652" watchObservedRunningTime="2026-07-09 16:49:47.304508631 +0000 UTC m=+2065.102729172" Jul 09 16:49:47.311423 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:47.311396 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:47.316176 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:47.316152 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:48.281766 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:48.281731 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:48.282685 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:48.282665 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:49:50.524020 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.523997 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:50.556453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556379 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgd2w\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-kube-api-access-fgd2w\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556425 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istiod-ca-cert\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-socket\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556710 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556476 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-podinfo\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556710 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556507 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-certs\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556710 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-credential-socket\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556710 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556564 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-data\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556710 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556645 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-token\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.556710 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556667 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-envoy\") pod \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\" (UID: \"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1\") " Jul 09 16:49:50.557036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556778 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:50.557036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556844 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:50.557036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.556981 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.557036 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.557000 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.557253 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.557053 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:50.557312 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.557263 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:50.557756 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.557725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:50.559114 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.559088 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:50.559252 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.559234 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:49:50.559305 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.559249 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-kube-api-access-fgd2w" (OuterVolumeSpecName: "kube-api-access-fgd2w") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "kube-api-access-fgd2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:50.559341 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.559300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" (UID: "8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:49:50.657732 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657699 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.657732 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657726 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.657732 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657735 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.657999 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657743 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgd2w\" (UniqueName: \"kubernetes.io/projected/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-kube-api-access-fgd2w\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.657999 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657751 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.657999 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657759 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:50.657999 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:50.657767 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:49:51.298090 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.298055 2562 generic.go:356] "Generic (PLEG): container finished" podID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerID="f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608" exitCode=0 Jul 09 16:49:51.298277 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.298100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" event={"ID":"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1","Type":"ContainerDied","Data":"f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608"} Jul 09 16:49:51.298277 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.298110 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" Jul 09 16:49:51.298277 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.298155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2" event={"ID":"8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1","Type":"ContainerDied","Data":"176c2d6150c36bcf4f60ec6bf47b80e424f77a2873beb4680c16d61e34dadcf2"} Jul 09 16:49:51.298277 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.298186 2562 scope.go:117] "RemoveContainer" containerID="f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608" Jul 09 16:49:51.306829 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.306811 2562 scope.go:117] "RemoveContainer" containerID="2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3" Jul 09 16:49:51.314281 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.314262 2562 scope.go:117] "RemoveContainer" containerID="f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608" Jul 09 16:49:51.314541 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:51.314521 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608\": container with ID starting with f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608 not found: ID does not exist" containerID="f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608" Jul 09 16:49:51.314594 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.314552 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608"} err="failed to get container status \"f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608\": rpc error: code = NotFound desc = could not find container \"f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608\": container with ID starting with f268eef34bedaeab5b45e03be8c4e6b1aab7ce587e5797925fbc3b0244b19608 not found: ID does not exist" Jul 09 16:49:51.314594 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.314570 2562 scope.go:117] "RemoveContainer" containerID="2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3" Jul 09 16:49:51.314784 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:49:51.314766 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3\": container with ID starting with 2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3 not found: ID does not exist" containerID="2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3" Jul 09 16:49:51.314830 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.314790 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3"} err="failed to get container status \"2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3\": rpc error: code = NotFound desc = could not find container \"2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3\": container with ID starting with 2c0cba7cc42fee690539c168a4c0b342fcf075564b0fbf2c1e7cbb7c5714afb3 not found: ID does not exist" Jul 09 16:49:51.320661 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.320639 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2"] Jul 09 16:49:51.327610 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:51.327585 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-554cc9-openshift-default-8669fbf655-ztlc2"] Jul 09 16:49:52.825636 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:52.825604 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" path="/var/lib/kubelet/pods/8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1/volumes" Jul 09 16:49:54.429755 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.429711 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64"] Jul 09 16:49:54.430274 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.430256 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" Jul 09 16:49:54.430344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.430278 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" Jul 09 16:49:54.430344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.430296 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" Jul 09 16:49:54.430344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.430304 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" Jul 09 16:49:54.430462 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.430394 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" Jul 09 16:49:54.430462 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.430407 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2ec65f-1d9b-4868-9c97-e00b7ad9f2e1" containerName="istio-proxy" Jul 09 16:49:54.433759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.433736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.438052 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.438033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6bf011-openshift-default-dockercfg-c26pp\"" Jul 09 16:49:54.446062 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.446035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64"] Jul 09 16:49:54.493769 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.493718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-token\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.493978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.493777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-credential-socket\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.493978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.493827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-certs\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.493978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.493861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-podinfo\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.493978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.493928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-data\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.493978 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.493961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94c88b6-519f-4179-ad9c-4fc951202a9a-istiod-ca-cert\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.494139 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.494006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhl5\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-kube-api-access-fmhl5\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.494139 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.494045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-socket\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.494139 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.494080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-envoy\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595123 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-credential-socket\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595123 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-certs\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595379 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-podinfo\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595379 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-data\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595379 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94c88b6-519f-4179-ad9c-4fc951202a9a-istiod-ca-cert\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595379 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhl5\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-kube-api-access-fmhl5\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595379 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-socket\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595379 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-envoy\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-token\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-credential-socket\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-certs\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595660 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-data\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595828 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595796 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-socket\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.595954 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.595936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94c88b6-519f-4179-ad9c-4fc951202a9a-istiod-ca-cert\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.597482 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.597460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-envoy\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.597670 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.597651 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-podinfo\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.603055 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.603025 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-token\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.603386 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.603365 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhl5\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-kube-api-access-fmhl5\") pod \"e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.745612 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.745511 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:54.886541 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.886514 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64"] Jul 09 16:49:54.888360 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:49:54.888333 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94c88b6_519f_4179_ad9c_4fc951202a9a.slice/crio-262bec5391b2152923eda0a4dcc828cb9efc21c89be6da0dbe502b3d2787a70f WatchSource:0}: Error finding container 262bec5391b2152923eda0a4dcc828cb9efc21c89be6da0dbe502b3d2787a70f: Status 404 returned error can't find the container with id 262bec5391b2152923eda0a4dcc828cb9efc21c89be6da0dbe502b3d2787a70f Jul 09 16:49:54.890493 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.890463 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:54.890561 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.890553 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:54.890601 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:54.890584 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:49:55.321039 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:55.321005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" event={"ID":"b94c88b6-519f-4179-ad9c-4fc951202a9a","Type":"ContainerStarted","Data":"90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a"} Jul 09 16:49:55.321232 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:55.321055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" event={"ID":"b94c88b6-519f-4179-ad9c-4fc951202a9a","Type":"ContainerStarted","Data":"262bec5391b2152923eda0a4dcc828cb9efc21c89be6da0dbe502b3d2787a70f"} Jul 09 16:49:55.348178 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:55.348124 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podStartSLOduration=1.348102925 podStartE2EDuration="1.348102925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:55.345306474 +0000 UTC m=+2073.143527036" watchObservedRunningTime="2026-07-09 16:49:55.348102925 +0000 UTC m=+2073.146323462" Jul 09 16:49:55.745972 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:55.745931 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:55.750936 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:55.750910 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:56.326344 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:56.326303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:49:56.327442 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:49:56.327423 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:50:04.323079 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:04.323028 2562 patch_prober.go:28] interesting pod/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 16:50:04.323470 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:04.323108 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 16:50:06.379953 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:06.379923 2562 generic.go:356] "Generic (PLEG): container finished" podID="35d6f697-1343-44d9-8912-c55786b4316b" containerID="35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607" exitCode=0 Jul 09 16:50:06.380382 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:06.379994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" event={"ID":"35d6f697-1343-44d9-8912-c55786b4316b","Type":"ContainerDied","Data":"35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607"} Jul 09 16:50:06.380426 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:06.380391 2562 scope.go:117] "RemoveContainer" containerID="35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607" Jul 09 16:50:06.381476 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:06.381441 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:50:06.381594 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:06.381525 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:50:06.381594 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:06.381560 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:50:07.386754 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:07.386722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" event={"ID":"35d6f697-1343-44d9-8912-c55786b4316b","Type":"ContainerStarted","Data":"3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98"} Jul 09 16:50:08.311847 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:08.311818 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:50:08.312149 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:08.312000 2562 patch_prober.go:28] interesting pod/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 09 16:50:08.312232 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:08.312185 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 09 16:50:09.312247 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:09.312206 2562 patch_prober.go:28] interesting pod/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 09 16:50:09.312717 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:09.312272 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 09 16:50:10.312421 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:10.312389 2562 patch_prober.go:28] interesting pod/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 09 16:50:10.312943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:10.312909 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 09 16:50:11.315736 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:11.315702 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:50:11.316176 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:11.316116 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:50:11.316975 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:11.316940 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:50:12.368114 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:12.368063 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 16:50:12.368513 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:12.368130 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 16:50:14.421038 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:14.421002 2562 generic.go:356] "Generic (PLEG): container finished" podID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerID="90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a" exitCode=0 Jul 09 16:50:14.421587 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:14.421045 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" event={"ID":"b94c88b6-519f-4179-ad9c-4fc951202a9a","Type":"ContainerDied","Data":"90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a"} Jul 09 16:50:14.421662 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:14.421590 2562 scope.go:117] "RemoveContainer" containerID="90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a" Jul 09 16:50:14.422599 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:14.422571 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:50:14.422704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:14.422653 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:50:14.422772 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:14.422702 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 16:50:15.428935 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:15.428873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" event={"ID":"b94c88b6-519f-4179-ad9c-4fc951202a9a","Type":"ContainerStarted","Data":"b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065"} Jul 09 16:50:15.747051 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:15.746925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:50:15.747458 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:15.747430 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 09 16:50:15.747535 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:15.747498 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 09 16:50:16.746591 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:16.746549 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 09 16:50:16.747229 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:16.747193 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 09 16:50:17.746470 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:17.746429 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 09 16:50:17.746763 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:17.746732 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 09 16:50:18.747112 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:18.747069 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 09 16:50:18.748153 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:18.747699 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 09 16:50:19.753258 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:19.753224 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:50:19.753655 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:19.753600 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:50:19.754488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:19.754468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:50:22.833271 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:22.833245 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:50:22.836544 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:50:22.836517 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:55:22.861502 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:55:22.861474 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:55:22.864517 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:55:22.864495 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8c2d_063206ae-2dba-4fe8-9def-da6101746a51/ovn-acl-logging/0.log" Jul 09 16:56:12.829874 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:12.829833 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs"] Jul 09 16:56:12.830401 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:12.830116 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" containerID="cri-o://3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98" gracePeriod=30 Jul 09 16:56:18.098671 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.098602 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:56:18.171744 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.171711 2562 generic.go:356] "Generic (PLEG): container finished" podID="35d6f697-1343-44d9-8912-c55786b4316b" containerID="3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98" exitCode=0 Jul 09 16:56:18.171948 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.171759 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" Jul 09 16:56:18.171948 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.171779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" event={"ID":"35d6f697-1343-44d9-8912-c55786b4316b","Type":"ContainerDied","Data":"3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98"} Jul 09 16:56:18.171948 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.171851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs" event={"ID":"35d6f697-1343-44d9-8912-c55786b4316b","Type":"ContainerDied","Data":"45ccfbde88904aa056d4f895e090e5fc6d310ac06d9c86c08fbfdcd77ee0f280"} Jul 09 16:56:18.171948 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.171906 2562 scope.go:117] "RemoveContainer" containerID="3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98" Jul 09 16:56:18.180778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.180759 2562 scope.go:117] "RemoveContainer" containerID="35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607" Jul 09 16:56:18.188950 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.188926 2562 scope.go:117] "RemoveContainer" containerID="3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98" Jul 09 16:56:18.189242 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:56:18.189221 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98\": container with ID starting with 3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98 not found: ID does not exist" containerID="3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98" Jul 09 16:56:18.189299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.189252 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98"} err="failed to get container status \"3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98\": rpc error: code = NotFound desc = could not find container \"3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98\": container with ID starting with 3874ab5e6b67d95c334b2c19fc031bedd843c6de649959d901f9f9f89b0d4e98 not found: ID does not exist" Jul 09 16:56:18.189299 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.189273 2562 scope.go:117] "RemoveContainer" containerID="35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607" Jul 09 16:56:18.189528 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:56:18.189509 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607\": container with ID starting with 35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607 not found: ID does not exist" containerID="35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607" Jul 09 16:56:18.189579 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.189536 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607"} err="failed to get container status \"35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607\": rpc error: code = NotFound desc = could not find container \"35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607\": container with ID starting with 35e4a8808ff267b2ade0c3ab81339adad0f97d3f2d50314da9a5650095b56607 not found: ID does not exist" Jul 09 16:56:18.284025 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.283977 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-envoy\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284205 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284051 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-socket\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284205 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284107 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d6f697-1343-44d9-8912-c55786b4316b-istiod-ca-cert\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284205 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284176 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d6f697-1343-44d9-8912-c55786b4316b-istio-podinfo\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284205 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284204 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-istio-token\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284404 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284236 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-credential-socket\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284404 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284260 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs9rj\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-kube-api-access-fs9rj\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284404 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284285 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-data\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284404 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284311 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-certs\") pod \"35d6f697-1343-44d9-8912-c55786b4316b\" (UID: \"35d6f697-1343-44d9-8912-c55786b4316b\") " Jul 09 16:56:18.284404 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284359 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:18.284668 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284562 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:18.284668 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284646 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.284668 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284649 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d6f697-1343-44d9-8912-c55786b4316b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:56:18.284872 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.284821 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:18.285411 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.285370 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-data" (OuterVolumeSpecName: "istio-data") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:18.286304 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.286278 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35d6f697-1343-44d9-8912-c55786b4316b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:56:18.286414 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.286363 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:18.286700 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.286685 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-kube-api-access-fs9rj" (OuterVolumeSpecName: "kube-api-access-fs9rj") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "kube-api-access-fs9rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:18.287002 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.286984 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-istio-token" (OuterVolumeSpecName: "istio-token") pod "35d6f697-1343-44d9-8912-c55786b4316b" (UID: "35d6f697-1343-44d9-8912-c55786b4316b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:18.385488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385393 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35d6f697-1343-44d9-8912-c55786b4316b-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385424 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385436 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385469 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9rj\" (UniqueName: \"kubernetes.io/projected/35d6f697-1343-44d9-8912-c55786b4316b-kube-api-access-fs9rj\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385479 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385488 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385488 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385814 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385497 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35d6f697-1343-44d9-8912-c55786b4316b-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.385814 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.385505 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35d6f697-1343-44d9-8912-c55786b4316b-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:18.497537 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.497506 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs"] Jul 09 16:56:18.503285 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.503256 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-bef56b-openshift-default-5569f78675-6h6cs"] Jul 09 16:56:18.820735 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:18.820700 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d6f697-1343-44d9-8912-c55786b4316b" path="/var/lib/kubelet/pods/35d6f697-1343-44d9-8912-c55786b4316b/volumes" Jul 09 16:56:19.736578 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:19.736544 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64"] Jul 09 16:56:19.737077 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:19.736744 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" containerID="cri-o://b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065" gracePeriod=30 Jul 09 16:56:19.755501 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:19.755445 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 16:56:19.755629 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:19.755510 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 16:56:24.993387 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:24.993356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:56:25.046757 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046665 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-token\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.046757 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046707 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-certs\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.046757 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046734 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-envoy\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047056 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046778 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-socket\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047056 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046856 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-data\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047056 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046942 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-credential-socket\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047056 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.046980 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-podinfo\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047056 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047029 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmhl5\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-kube-api-access-fmhl5\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047074 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94c88b6-519f-4179-ad9c-4fc951202a9a-istiod-ca-cert\") pod \"b94c88b6-519f-4179-ad9c-4fc951202a9a\" (UID: \"b94c88b6-519f-4179-ad9c-4fc951202a9a\") " Jul 09 16:56:25.047338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047075 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:25.047338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047120 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:25.047495 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047398 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.047495 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047423 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.047495 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047431 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:25.047738 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047718 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94c88b6-519f-4179-ad9c-4fc951202a9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:56:25.047944 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.047920 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:25.049185 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.049157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 16:56:25.049290 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.049253 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:25.049346 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.049329 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-kube-api-access-fmhl5" (OuterVolumeSpecName: "kube-api-access-fmhl5") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "kube-api-access-fmhl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:25.049418 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.049402 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b94c88b6-519f-4179-ad9c-4fc951202a9a" (UID: "b94c88b6-519f-4179-ad9c-4fc951202a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:25.149000 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.148961 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.149000 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.148992 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.149000 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.149003 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.149244 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.149012 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmhl5\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-kube-api-access-fmhl5\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.149244 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.149023 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94c88b6-519f-4179-ad9c-4fc951202a9a-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.149244 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.149030 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.149244 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.149038 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94c88b6-519f-4179-ad9c-4fc951202a9a-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\"" Jul 09 16:56:25.212453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.212416 2562 generic.go:356] "Generic (PLEG): container finished" podID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerID="b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065" exitCode=0 Jul 09 16:56:25.212633 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.212470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" event={"ID":"b94c88b6-519f-4179-ad9c-4fc951202a9a","Type":"ContainerDied","Data":"b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065"} Jul 09 16:56:25.212633 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.212480 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" Jul 09 16:56:25.212633 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.212525 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64" event={"ID":"b94c88b6-519f-4179-ad9c-4fc951202a9a","Type":"ContainerDied","Data":"262bec5391b2152923eda0a4dcc828cb9efc21c89be6da0dbe502b3d2787a70f"} Jul 09 16:56:25.212633 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.212547 2562 scope.go:117] "RemoveContainer" containerID="b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065" Jul 09 16:56:25.221732 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.221711 2562 scope.go:117] "RemoveContainer" containerID="90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a" Jul 09 16:56:25.231048 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.231025 2562 scope.go:117] "RemoveContainer" containerID="b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065" Jul 09 16:56:25.231311 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:56:25.231291 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065\": container with ID starting with b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065 not found: ID does not exist" containerID="b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065" Jul 09 16:56:25.231361 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.231320 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065"} err="failed to get container status \"b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065\": rpc error: code = NotFound desc = could not find container \"b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065\": container with ID starting with b73ba620bc89d2a969d6369f8a45ceb0e39758edd6b0ae3b7030227e3e037065 not found: ID does not exist" Jul 09 16:56:25.231361 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.231341 2562 scope.go:117] "RemoveContainer" containerID="90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a" Jul 09 16:56:25.231602 ip-10-0-140-120 kubenswrapper[2562]: E0709 16:56:25.231579 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a\": container with ID starting with 90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a not found: ID does not exist" containerID="90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a" Jul 09 16:56:25.231656 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.231615 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a"} err="failed to get container status \"90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a\": rpc error: code = NotFound desc = could not find container \"90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a\": container with ID starting with 90dd432723bdc2aaa6ebe2046af41e977608bb0ace79be0bd8f75dc9ea86471a not found: ID does not exist" Jul 09 16:56:25.236307 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.236280 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64"] Jul 09 16:56:25.241196 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.241174 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6bf011-openshift-default-86c6798d99-f4b64"] Jul 09 16:56:25.521916 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.521871 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mqgzq_647bf3ec-295b-4e39-b3c8-f98c2593d541/manager/0.log" Jul 09 16:56:25.644283 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:25.644248 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zt9cn_8d6c4681-bee6-4871-82e0-8642bfba1482/manager/0.log" Jul 09 16:56:26.387898 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:26.387863 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-tr5kr_016b5be6-0fcf-4325-ab16-c0063ba67258/copy-manifests/0.log" Jul 09 16:56:26.394807 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:26.394788 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-tr5kr_016b5be6-0fcf-4325-ab16-c0063ba67258/manager/0.log" Jul 09 16:56:26.820082 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:26.820051 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" path="/var/lib/kubelet/pods/b94c88b6-519f-4179-ad9c-4fc951202a9a/volumes" Jul 09 16:56:28.284746 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:28.284712 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-77sd6_c505acff-5ef8-4bd7-9d7b-522e27721902/manager/0.log" Jul 09 16:56:28.750260 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:28.750223 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-x27xk_ba7b15aa-e4cb-45b3-b3f6-210b14ed155d/manager/0.log" Jul 09 16:56:29.345411 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:29.345384 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jcbx5_7196b52a-4961-455d-9144-1795e9eb0dd7/istio-proxy/0.log" Jul 09 16:56:29.465125 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:29.465090 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-n7lbj_4018af85-7af5-4424-8b76-b74aa59c4b82/discovery/0.log" Jul 09 16:56:29.581943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:29.581911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d9878b65-pcrrl_6c653629-ab5b-4dd5-a29c-5384f723cb7d/kube-auth-proxy/0.log" Jul 09 16:56:29.832463 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:29.832437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/1.log" Jul 09 16:56:30.546232 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:30.546202 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xthf5_89bd19da-4752-45e7-b8ea-514196e34df0/main/0.log" Jul 09 16:56:31.015546 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:31.015512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6t9fv_df96aa47-bd8f-4732-8039-760050e92e56/main/0.log" Jul 09 16:56:37.952167 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:37.952140 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b8dkn_8b1451f3-2475-4614-b6a8-df41425e93b8/global-pull-secret-syncer/0.log" Jul 09 16:56:38.128370 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:38.128342 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wpqxt_197126bf-b934-4bb7-b1a4-d30338ffc52e/konnectivity-agent/0.log" Jul 09 16:56:38.250943 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:38.250847 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-120.ec2.internal_09061521089a77af38194f9d3efcc858/haproxy/0.log" Jul 09 16:56:43.213699 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:43.213666 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-77sd6_c505acff-5ef8-4bd7-9d7b-522e27721902/manager/0.log" Jul 09 16:56:43.418766 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:43.418732 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-x27xk_ba7b15aa-e4cb-45b3-b3f6-210b14ed155d/manager/0.log" Jul 09 16:56:45.312762 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.312731 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8v9nk_753f2dd3-dccf-4251-9616-6a17b8d3b805/kube-state-metrics/0.log" Jul 09 16:56:45.349578 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.349549 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8v9nk_753f2dd3-dccf-4251-9616-6a17b8d3b805/kube-rbac-proxy-main/0.log" Jul 09 16:56:45.380993 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.380968 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-8v9nk_753f2dd3-dccf-4251-9616-6a17b8d3b805/kube-rbac-proxy-self/0.log" Jul 09 16:56:45.483530 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.483499 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-h82rf_dc772d81-4f42-460b-93f3-286deb1a4547/monitoring-plugin/0.log" Jul 09 16:56:45.710134 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.710105 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdvsm_9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3/node-exporter/0.log" Jul 09 16:56:45.735283 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.735251 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdvsm_9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3/kube-rbac-proxy/0.log" Jul 09 16:56:45.759937 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.759913 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdvsm_9d7c93d8-c4ea-41ad-bad5-3b4d30dadbd3/init-textfile/0.log" Jul 09 16:56:45.807912 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.807866 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-h954l_a08ce60f-1817-43ba-9c14-0c427fac6444/kube-rbac-proxy-main/0.log" Jul 09 16:56:45.837198 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.837170 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-h954l_a08ce60f-1817-43ba-9c14-0c427fac6444/kube-rbac-proxy-self/0.log" Jul 09 16:56:45.867697 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.867672 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-h954l_a08ce60f-1817-43ba-9c14-0c427fac6444/openshift-state-metrics/0.log" Jul 09 16:56:45.910955 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.910929 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/prometheus/0.log" Jul 09 16:56:45.935463 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.935413 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/config-reloader/0.log" Jul 09 16:56:45.959045 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.959023 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/thanos-sidecar/0.log" Jul 09 16:56:45.982435 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:45.982414 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/kube-rbac-proxy-web/0.log" Jul 09 16:56:46.006551 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:46.006534 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/kube-rbac-proxy/0.log" Jul 09 16:56:46.030705 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:46.030686 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/kube-rbac-proxy-thanos/0.log" Jul 09 16:56:46.054489 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:46.054455 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a2bf0ec-2631-4108-92aa-082125631266/init-config-reloader/0.log" Jul 09 16:56:46.203220 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:46.203132 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-zmz44_88a078a0-4ac5-4075-aa8c-40e05e10b015/prometheus-operator-admission-webhook/0.log" Jul 09 16:56:47.666778 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.666742 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5"] Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667142 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667154 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667164 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667169 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667182 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667189 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667197 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" Jul 09 16:56:47.667213 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667203 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" Jul 09 16:56:47.667451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667267 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" Jul 09 16:56:47.667451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667277 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d6f697-1343-44d9-8912-c55786b4316b" containerName="istio-proxy" Jul 09 16:56:47.667451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667284 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" Jul 09 16:56:47.667451 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.667291 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94c88b6-519f-4179-ad9c-4fc951202a9a" containerName="istio-proxy" Jul 09 16:56:47.670495 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.670472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.673759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.673740 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b7vxd\"/\"default-dockercfg-w4l5d\"" Jul 09 16:56:47.673759 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.673748 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7vxd\"/\"openshift-service-ca.crt\"" Jul 09 16:56:47.675281 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.675252 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7vxd\"/\"kube-root-ca.crt\"" Jul 09 16:56:47.678058 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.677928 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5"] Jul 09 16:56:47.756423 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.756377 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-sys\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.756423 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.756424 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwjg\" (UniqueName: \"kubernetes.io/projected/3e50936a-122f-42ee-9fe1-c6274013b07e-kube-api-access-dqwjg\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.756662 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.756523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-podres\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.756662 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.756611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-proc\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.756662 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.756654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-lib-modules\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857203 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-lib-modules\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-sys\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwjg\" (UniqueName: \"kubernetes.io/projected/3e50936a-122f-42ee-9fe1-c6274013b07e-kube-api-access-dqwjg\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-podres\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-proc\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-lib-modules\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-sys\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857400 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-proc\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.857646 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.857422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e50936a-122f-42ee-9fe1-c6274013b07e-podres\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.871456 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.871426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwjg\" (UniqueName: \"kubernetes.io/projected/3e50936a-122f-42ee-9fe1-c6274013b07e-kube-api-access-dqwjg\") pod \"perf-node-gather-daemonset-jr6r5\" (UID: \"3e50936a-122f-42ee-9fe1-c6274013b07e\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:47.981484 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:47.981398 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:48.117388 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:48.117364 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5"] Jul 09 16:56:48.119940 ip-10-0-140-120 kubenswrapper[2562]: W0709 16:56:48.119913 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e50936a_122f_42ee_9fe1_c6274013b07e.slice/crio-0322a4c1321051991624afb2f79a0d511dee188b453b4a728c15f2443da484b6 WatchSource:0}: Error finding container 0322a4c1321051991624afb2f79a0d511dee188b453b4a728c15f2443da484b6: Status 404 returned error can't find the container with id 0322a4c1321051991624afb2f79a0d511dee188b453b4a728c15f2443da484b6 Jul 09 16:56:48.320654 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:48.320619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" event={"ID":"3e50936a-122f-42ee-9fe1-c6274013b07e","Type":"ContainerStarted","Data":"c27610a3c12ff50d04067f3ffcdb07cff47b58a563320185bc9294191410d8e2"} Jul 09 16:56:48.320832 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:48.320678 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" event={"ID":"3e50936a-122f-42ee-9fe1-c6274013b07e","Type":"ContainerStarted","Data":"0322a4c1321051991624afb2f79a0d511dee188b453b4a728c15f2443da484b6"} Jul 09 16:56:48.320832 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:48.320746 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:48.337582 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:48.337532 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" podStartSLOduration=1.3375171940000001 podStartE2EDuration="1.337517194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:48.336609819 +0000 UTC m=+2486.134830364" watchObservedRunningTime="2026-07-09 16:56:48.337517194 +0000 UTC m=+2486.135737735" Jul 09 16:56:50.617218 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:50.617188 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clfl2_9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb/dns/0.log" Jul 09 16:56:50.642704 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:50.642677 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clfl2_9bb89b1d-0dc3-4515-b11a-9646e3f5ebcb/kube-rbac-proxy/0.log" Jul 09 16:56:50.746639 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:50.746610 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d6m5b_25c4651e-3a67-4a78-a4b0-57598fb5f5b0/dns-node-resolver/0.log" Jul 09 16:56:51.329453 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:51.329419 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f68bbf98-jfs56_d4a854d5-5647-4ac3-9d40-bb1d99832b48/registry/0.log" Jul 09 16:56:51.400033 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:51.400005 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xbgpm_cc3f0165-edbd-404e-9ca9-ca0c82314028/node-ca/0.log" Jul 09 16:56:52.335487 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:52.335456 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jcbx5_7196b52a-4961-455d-9144-1795e9eb0dd7/istio-proxy/0.log" Jul 09 16:56:52.445954 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:52.445920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-n7lbj_4018af85-7af5-4424-8b76-b74aa59c4b82/discovery/0.log" Jul 09 16:56:52.466548 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:52.466515 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d9878b65-pcrrl_6c653629-ab5b-4dd5-a29c-5384f723cb7d/kube-auth-proxy/0.log" Jul 09 16:56:52.522156 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:52.522125 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/0.log" Jul 09 16:56:52.552836 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:52.552805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/1.log" Jul 09 16:56:53.384373 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:53.384343 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rx2z5_26741143-5141-49b1-96a9-7f82db5af8a7/serve-healthcheck-canary/0.log" Jul 09 16:56:54.037777 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:54.037746 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z8bns_81464632-b149-4ecf-b185-c77649824390/kube-rbac-proxy/0.log" Jul 09 16:56:54.059414 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:54.059384 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z8bns_81464632-b149-4ecf-b185-c77649824390/exporter/0.log" Jul 09 16:56:54.083132 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:54.083105 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z8bns_81464632-b149-4ecf-b185-c77649824390/extractor/0.log" Jul 09 16:56:54.336074 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:54.336002 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-jr6r5" Jul 09 16:56:56.166106 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:56.166068 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mqgzq_647bf3ec-295b-4e39-b3c8-f98c2593d541/manager/0.log" Jul 09 16:56:56.300670 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:56.300617 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zt9cn_8d6c4681-bee6-4871-82e0-8642bfba1482/manager/0.log" Jul 09 16:56:56.704765 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:56.704738 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-tr5kr_016b5be6-0fcf-4325-ab16-c0063ba67258/manager/0.log" Jul 09 16:56:56.725781 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:56.725758 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-tr5kr_016b5be6-0fcf-4325-ab16-c0063ba67258/copy-manifests/0.log" Jul 09 16:56:58.050192 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:58.050164 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5x5rp_392bca38-51fa-4cbc-9db0-b6dbf77712d7/openshift-lws-operator/0.log" Jul 09 16:56:58.694545 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:58.694518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jcbx5_7196b52a-4961-455d-9144-1795e9eb0dd7/istio-proxy/0.log" Jul 09 16:56:58.800424 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:58.800392 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-n7lbj_4018af85-7af5-4424-8b76-b74aa59c4b82/discovery/0.log" Jul 09 16:56:58.811155 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:58.811123 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d9878b65-pcrrl_6c653629-ab5b-4dd5-a29c-5384f723cb7d/kube-auth-proxy/0.log" Jul 09 16:56:58.849514 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:58.849431 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/0.log" Jul 09 16:56:58.881187 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:56:58.881158 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/1.log" Jul 09 16:57:00.108358 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:00.108276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678jcbx5_7196b52a-4961-455d-9144-1795e9eb0dd7/istio-proxy/0.log" Jul 09 16:57:00.210342 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:00.210307 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-n7lbj_4018af85-7af5-4424-8b76-b74aa59c4b82/discovery/0.log" Jul 09 16:57:00.224083 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:00.224054 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59d9878b65-pcrrl_6c653629-ab5b-4dd5-a29c-5384f723cb7d/kube-auth-proxy/0.log" Jul 09 16:57:00.266263 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:00.266230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/0.log" Jul 09 16:57:00.295512 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:00.295477 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-btwv2_25b10fa6-eb53-417a-8f52-79b9407df75d/istio-proxy/1.log" Jul 09 16:57:05.111791 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.111762 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/kube-multus-additional-cni-plugins/0.log" Jul 09 16:57:05.136823 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.136793 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/egress-router-binary-copy/0.log" Jul 09 16:57:05.159241 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.159215 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/cni-plugins/0.log" Jul 09 16:57:05.182989 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.182963 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/bond-cni-plugin/0.log" Jul 09 16:57:05.204585 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.204552 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/routeoverride-cni/0.log" Jul 09 16:57:05.226338 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.226307 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/whereabouts-cni-bincopy/0.log" Jul 09 16:57:05.248794 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.248767 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hrjzl_58bdbaff-323b-49e4-a33c-3ba51e4ed9a6/whereabouts-cni/0.log" Jul 09 16:57:05.280159 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.280132 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k8xpd_a61dae83-f0ff-46cd-86f4-4a898c056602/kube-multus/0.log" Jul 09 16:57:05.473303 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.473276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mpbbn_8d7c05a8-7911-461d-a206-57df68595e5e/network-metrics-daemon/0.log" Jul 09 16:57:05.493196 ip-10-0-140-120 kubenswrapper[2562]: I0709 16:57:05.493157 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mpbbn_8d7c05a8-7911-461d-a206-57df68595e5e/kube-rbac-proxy/0.log"