Aug 04 19:13:33.447789 ip-10-0-143-62 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 19:13:33.447801 ip-10-0-143-62 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 19:13:33.447811 ip-10-0-143-62 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 19:13:33.448148 ip-10-0-143-62 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 19:13:43.691038 ip-10-0-143-62 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 19:13:43.691058 ip-10-0-143-62 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot df962a2719b04acbb6a29743ea43f63d -- Aug 04 19:15:22.928659 ip-10-0-143-62 systemd[1]: Starting Kubernetes Kubelet... Aug 04 19:15:23.380497 ip-10-0-143-62 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:23.380497 ip-10-0-143-62 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 19:15:23.380497 ip-10-0-143-62 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:23.380497 ip-10-0-143-62 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 19:15:23.380497 ip-10-0-143-62 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 19:15:23.382079 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.381953 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 19:15:23.389190 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389161 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:23.389190 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389182 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:23.389190 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389187 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:23.389190 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389193 2579 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:23.389190 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389197 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389201 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389205 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389209 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389214 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389217 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389221 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389224 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389227 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389231 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389233 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389238 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389241 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389244 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389247 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389250 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389253 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389255 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389258 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:23.389371 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389261 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389266 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389271 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389274 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389276 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389279 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389282 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389285 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389287 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389290 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389294 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389297 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389299 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389302 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389305 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389308 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389312 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389316 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389318 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389321 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:23.389840 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389337 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389341 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389346 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389350 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389354 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389356 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389359 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389362 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389364 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389369 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389372 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389375 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389378 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389381 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389385 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389388 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389391 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389394 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389396 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:23.390357 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389400 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389403 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389406 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389409 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389412 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389414 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389417 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389420 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389422 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389425 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389427 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389431 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389433 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389436 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389438 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389441 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389443 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389446 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389448 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389451 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:23.390829 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389453 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389456 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389458 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389461 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389463 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389467 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389470 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389474 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389477 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389479 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389482 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389485 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389488 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389491 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389494 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389497 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:23.391370 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.389500 2579 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:23.391809 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391794 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:23.391809 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391809 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391813 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391817 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391820 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391824 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391828 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391834 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391837 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391840 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391844 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391846 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391850 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391853 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391855 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391858 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391861 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391864 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391867 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391870 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:23.391864 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391873 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391876 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391879 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391881 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391884 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391886 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391889 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391892 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391894 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391897 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391899 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391903 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391905 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391907 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391910 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391912 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391915 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391917 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391921 2579 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391923 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:23.392348 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391926 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391929 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391932 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391935 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391937 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391940 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391943 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391945 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391948 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391950 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391954 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391957 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391960 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391964 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391969 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391972 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391975 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391978 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391981 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:23.392847 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391983 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391986 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391988 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391991 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391994 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391996 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.391999 2579 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392001 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392004 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392007 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392009 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392011 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392014 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392016 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392020 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392022 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392025 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392028 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392030 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392033 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:23.393351 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392036 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392038 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392041 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392043 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392046 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392049 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392051 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392055 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392058 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392060 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392063 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392066 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392069 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392071 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392074 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392076 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392079 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392082 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392084 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392087 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:23.393855 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392501 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392507 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392510 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392512 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392515 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392518 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392522 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392524 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392527 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392530 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392533 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392535 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392538 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392541 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392543 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392546 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392548 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392551 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392553 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392557 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:23.394362 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392560 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392562 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392565 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392568 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392571 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392573 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392576 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392579 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392581 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392584 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392586 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392589 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392592 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392594 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392596 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392599 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392602 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392604 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392607 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392610 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:23.394914 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392612 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392615 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392617 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392620 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392622 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392625 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392627 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392630 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392633 2579 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392635 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392637 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392641 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392644 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392647 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392649 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392652 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392654 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392657 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392660 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:23.395413 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392662 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392664 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392667 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392670 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392672 2579 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392675 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392677 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392681 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392684 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392686 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392689 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392691 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392694 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392698 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392701 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392703 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392706 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392708 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392711 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392713 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:23.395878 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392716 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392718 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392721 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392723 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392729 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392732 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392735 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392737 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392740 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392743 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392748 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392751 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392754 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392758 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392762 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392766 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392769 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392772 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:23.396423 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392774 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.392777 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392850 2579 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392859 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392865 2579 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392870 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392874 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392878 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392883 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392888 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392891 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392894 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392898 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392901 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392904 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392907 2579 flags.go:64] FLAG: --cgroup-root="" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392910 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392914 2579 flags.go:64] FLAG: --client-ca-file="" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392917 2579 flags.go:64] FLAG: --cloud-provider="external" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392921 2579 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392935 2579 flags.go:64] FLAG: --cluster-domain="" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392938 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392941 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392944 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 19:15:23.396874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392948 2579 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392956 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392959 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392963 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392966 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392970 2579 flags.go:64] FLAG: --contention-profiling="false" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392972 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392975 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392979 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392982 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392986 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392989 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392992 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392995 2579 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.392998 2579 flags.go:64] FLAG: --enable-server="true" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393001 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393005 2579 flags.go:64] FLAG: --event-burst="100" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393009 2579 flags.go:64] FLAG: --event-qps="50" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393013 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393016 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393019 2579 flags.go:64] FLAG: --eviction-hard="" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393022 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393025 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393029 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393032 2579 flags.go:64] FLAG: --eviction-soft="" Aug 04 19:15:23.397468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393035 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393038 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393041 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393044 2579 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393047 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393050 2579 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393053 2579 flags.go:64] FLAG: --feature-gates="" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393057 2579 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393060 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393063 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393066 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393069 2579 flags.go:64] FLAG: --healthz-port="10248" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393072 2579 flags.go:64] FLAG: --help="false" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393075 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393078 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393081 2579 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393085 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393088 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393091 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393094 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393097 2579 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393100 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393103 2579 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 19:15:23.398071 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393106 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393110 2579 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393113 2579 flags.go:64] FLAG: --kube-reserved="" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393116 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393119 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393122 2579 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393125 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393128 2579 flags.go:64] FLAG: --lock-file="" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393130 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393134 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393137 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393143 2579 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393146 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393148 2579 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393151 2579 flags.go:64] FLAG: --logging-format="text" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393154 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393157 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393160 2579 flags.go:64] FLAG: --manifest-url="" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393163 2579 flags.go:64] FLAG: --manifest-url-header="" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393167 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393171 2579 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393175 2579 flags.go:64] FLAG: --max-pods="110" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393178 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393180 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393184 2579 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 19:15:23.398685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393187 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393189 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393192 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393195 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393202 2579 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393205 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393208 2579 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393212 2579 flags.go:64] FLAG: --pod-cidr="" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393214 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393220 2579 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393224 2579 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393227 2579 flags.go:64] FLAG: --pods-per-core="0" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393230 2579 flags.go:64] FLAG: --port="10250" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393233 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393236 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac278a437da0865d" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393239 2579 flags.go:64] FLAG: --qos-reserved="" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393242 2579 flags.go:64] FLAG: --read-only-port="10255" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393245 2579 flags.go:64] FLAG: --register-node="true" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393248 2579 flags.go:64] FLAG: --register-with-taints="" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393252 2579 flags.go:64] FLAG: --registry-burst="10" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393255 2579 flags.go:64] FLAG: --registry-qps="5" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393258 2579 flags.go:64] FLAG: --reserved-cpus="" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393260 2579 flags.go:64] FLAG: --reserved-memory="" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393264 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393267 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 19:15:23.399344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393270 2579 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393273 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393276 2579 flags.go:64] FLAG: --runonce="false" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393279 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393282 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393285 2579 flags.go:64] FLAG: --seccomp-default="false" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393288 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393291 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393294 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393297 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393300 2579 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393303 2579 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393305 2579 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393308 2579 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393311 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393314 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393318 2579 flags.go:64] FLAG: --system-cgroups="" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393321 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393342 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393345 2579 flags.go:64] FLAG: --tls-cert-file="" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393348 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393355 2579 flags.go:64] FLAG: --tls-min-version="" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393359 2579 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393362 2579 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393365 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 19:15:23.400121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393368 2579 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393371 2579 flags.go:64] FLAG: --v="2" Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393375 2579 flags.go:64] FLAG: --version="false" Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393380 2579 flags.go:64] FLAG: --vmodule="" Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393384 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393387 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393506 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393511 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393515 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393518 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393521 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393524 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393527 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393533 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393536 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393540 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393542 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393546 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393549 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393552 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393555 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:23.401076 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393558 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393561 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393564 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393567 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393570 2579 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393573 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393576 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393580 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393583 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393586 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393588 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393591 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393594 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393596 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393599 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393601 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393604 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393607 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393609 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393612 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:23.401704 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393614 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393617 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393620 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393623 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393627 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393629 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393632 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393635 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393637 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393641 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393645 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393648 2579 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393651 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393654 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393657 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393659 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393662 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393665 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393668 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:23.402200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393672 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393675 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393677 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393680 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393682 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393685 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393688 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393690 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393693 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393695 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393698 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393700 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393703 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393705 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393708 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393710 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393713 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393717 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393720 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393722 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:23.402697 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393725 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393728 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393730 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393733 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393736 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393740 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393743 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393746 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393749 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393751 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393754 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393757 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393761 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393764 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393766 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393769 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393772 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393774 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393777 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:23.403182 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393779 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393782 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393785 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393787 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393790 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.393793 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.393798 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.402194 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.402223 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.403053 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.403062 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403149 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403157 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403161 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:23.403811 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403164 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403167 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403170 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403172 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403175 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403178 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403181 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403184 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403187 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403190 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403193 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403196 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403199 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403202 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403205 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403207 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403210 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403212 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403216 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:23.404161 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403218 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403221 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403224 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403227 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403230 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403232 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403235 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403238 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403243 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403247 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403250 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403253 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403256 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403259 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403262 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403265 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403268 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403271 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403273 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:23.404632 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403276 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403279 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403281 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403284 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403286 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403289 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403292 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403294 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403297 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403300 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403302 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403305 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403308 2579 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403310 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403313 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403315 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403318 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403320 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403323 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403341 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:23.405100 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403346 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403351 2579 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403354 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403356 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403359 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403362 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403365 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403367 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403370 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403373 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403376 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403382 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403385 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403388 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403391 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403395 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403397 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403400 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403404 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:23.405610 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403407 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403410 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403412 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403415 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403418 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403420 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403423 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403425 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403428 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403431 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403434 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403436 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403439 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403443 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403445 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403448 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403451 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403454 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:23.406071 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403456 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.403462 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403587 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403593 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403597 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403601 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403604 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403608 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403611 2579 feature_gate.go:328] unrecognized feature gate: Example Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403614 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403617 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403619 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403623 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403626 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403629 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 19:15:23.406540 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403632 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403634 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403638 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403641 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403643 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403646 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403649 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403651 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403654 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403656 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403659 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403662 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403664 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403667 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403670 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403673 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403676 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403679 2579 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403681 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403684 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 19:15:23.406960 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403687 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403689 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403692 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403695 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403697 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403700 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403702 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403705 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403708 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403711 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403714 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403717 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403720 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403723 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403726 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403728 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403731 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403734 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403738 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 19:15:23.407566 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403741 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403744 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403747 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403749 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403752 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403754 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403757 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403760 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403763 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403765 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403768 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403771 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403773 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403777 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403780 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403783 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403786 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403790 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403793 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403796 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 19:15:23.408042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403799 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403802 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403805 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403808 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403810 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403813 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403817 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403820 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403823 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403826 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403829 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403832 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403835 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403837 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403840 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403843 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403845 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403848 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403851 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403854 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 19:15:23.408562 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403857 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403859 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403862 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403865 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403868 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403871 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:23.403873 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.403879 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.404018 2579 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.406301 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.407845 2579 server.go:1019] "Starting client certificate rotation" Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.407949 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 19:15:23.409044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.407989 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 19:15:23.411058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.411042 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 19:15:23.413255 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.413231 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 19:15:23.420420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.420394 2579 log.go:25] "Validated CRI v1 runtime API" Aug 04 19:15:23.422828 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.422806 2579 log.go:25] "Validated CRI v1 image API" Aug 04 19:15:23.425959 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.425937 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 19:15:23.429064 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.429038 2579 fs.go:135] Filesystem UUIDs: map[07208f8b-727b-416c-a3bb-aec32a0c14fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ce2cbf01-3284-472f-9cd7-fc9c1a932f0c:/dev/nvme0n1p3] Aug 04 19:15:23.429064 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.429059 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 19:15:23.436685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.435895 2579 manager.go:216] Machine: {Timestamp:2026-08-04 19:15:23.434472603 +0000 UTC m=+0.375269339 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117742 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b91498a0910f54a3a1d6de895f098 SystemUUID:ec2b9149-8a09-10f5-4a3a-1d6de895f098 BootID:df962a27-19b0-4acb-b6a2-9743ea43f63d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:bd:0f:50:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:bd:0f:50:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:13:f9:36:06:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 19:15:23.436952 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.436836 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 19:15:23.437051 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.437028 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 19:15:23.440814 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.440756 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 19:15:23.440984 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.440965 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 19:15:23.441018 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.440822 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 19:15:23.442286 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.442273 2579 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 19:15:23.442323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.442290 2579 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 19:15:23.442323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.442298 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 19:15:23.442323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.442313 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 19:15:23.443489 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.443476 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 19:15:23.447459 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447440 2579 state_mem.go:36] "Initialized new in-memory state store" Aug 04 19:15:23.447616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447606 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 19:15:23.447655 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447644 2579 kubelet.go:486] "Attempting to sync node with API server" Aug 04 19:15:23.447685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447657 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 19:15:23.447685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447681 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 19:15:23.447743 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447692 2579 kubelet.go:398] "Adding apiserver pod source" Aug 04 19:15:23.447743 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.447704 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 19:15:23.448161 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.448147 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 19:15:23.448265 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.448167 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 19:15:23.450781 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.450764 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 19:15:23.453186 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.453172 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 19:15:23.453252 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.453203 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 19:15:23.454871 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454859 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454876 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454882 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454888 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454897 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454902 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454911 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 19:15:23.454920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454917 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 19:15:23.455135 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454924 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 19:15:23.455135 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454930 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 19:15:23.455135 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454938 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 19:15:23.455135 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.454944 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 19:15:23.455735 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.455725 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 19:15:23.455735 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.455735 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 19:15:23.459065 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.459050 2579 server.go:1268] "Started kubelet" Aug 04 19:15:23.459159 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.459130 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 19:15:23.459642 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.459614 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 19:15:23.459916 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.459885 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 19:15:23.460029 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.460002 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 19:15:23.460081 ip-10-0-143-62 systemd[1]: Started Kubernetes Kubelet. Aug 04 19:15:23.460664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.459964 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 19:15:23.460664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.460156 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 19:15:23.461394 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.461379 2579 server.go:310] "Adding debug handlers to kubelet server" Aug 04 19:15:23.461682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.461657 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 19:15:23.463499 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.463479 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 19:15:23.466455 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.465425 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-62.ec2.internal.18c8b02497794376 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-62.ec2.internal,UID:ip-10-0-143-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-62.ec2.internal,},FirstTimestamp:2026-08-04 19:15:23.459019638 +0000 UTC m=+0.399816376,LastTimestamp:2026-08-04 19:15:23.459019638 +0000 UTC m=+0.399816376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-62.ec2.internal,}" Aug 04 19:15:23.467597 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.467584 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 19:15:23.467597 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.467589 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 19:15:23.468139 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.467969 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:23.468455 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468440 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 19:15:23.468455 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468456 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 19:15:23.468595 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468494 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 19:15:23.468650 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468596 2579 factory.go:55] Registering systemd factory Aug 04 19:15:23.468650 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468613 2579 factory.go:223] Registration of the systemd container factory successfully Aug 04 19:15:23.468746 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468597 2579 reconstruct.go:97] "Volume reconstruction finished" Aug 04 19:15:23.468746 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.468720 2579 reconciler.go:29] "Reconciler: start to sync state" Aug 04 19:15:23.469407 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.469392 2579 factory.go:153] Registering CRI-O factory Aug 04 19:15:23.469463 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.469412 2579 factory.go:223] Registration of the crio container factory successfully Aug 04 19:15:23.469463 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.469459 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 19:15:23.469532 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.469490 2579 factory.go:103] Registering Raw factory Aug 04 19:15:23.469532 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.469502 2579 manager.go:1196] Started watching for new ooms in manager Aug 04 19:15:23.470094 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.470081 2579 manager.go:349] Starting recovery of all containers Aug 04 19:15:23.479558 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.479503 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 19:15:23.479558 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.479507 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 19:15:23.483597 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.483579 2579 manager.go:354] Recovery completed Aug 04 19:15:23.487825 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.487802 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zgpd" Aug 04 19:15:23.488055 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.488043 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.489926 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.489905 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.490035 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.489939 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.490035 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.489951 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.490596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.490578 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 19:15:23.490596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.490592 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.490736 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.490594 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 19:15:23.490736 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.490632 2579 state_mem.go:36] "Initialized new in-memory state store" Aug 04 19:15:23.491305 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.491291 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.491397 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.491319 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.491397 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.491347 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.491778 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.491764 2579 policy_none.go:49] "None policy: Start" Aug 04 19:15:23.491778 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.491780 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 19:15:23.491881 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.491790 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 19:15:23.492647 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.492633 2579 policy_none.go:47] "Start" Aug 04 19:15:23.493780 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.493764 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zgpd" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532028 2579 manager.go:337] "Starting Device Plugin manager" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.532227 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532238 2579 server.go:87] "Starting device plugin registration server" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532568 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532579 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532675 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532767 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.532776 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.533396 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 19:15:23.533569 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.533447 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:23.602494 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.602459 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 19:15:23.603626 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.603604 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 19:15:23.603741 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.603645 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 19:15:23.603741 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.603668 2579 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 19:15:23.603741 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.603702 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 19:15:23.605870 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.605846 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:23.633563 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.633487 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.634497 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.634481 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.634596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.634510 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.634596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.634520 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.634596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.634567 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.646030 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.646001 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.646112 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.646038 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-62.ec2.internal\": node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:23.669964 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.669931 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:23.704066 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.704029 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal"] Aug 04 19:15:23.704184 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.704118 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.705707 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.705691 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.705793 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.705726 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.705793 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.705736 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.706843 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.706830 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.706944 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.706929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.706987 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.706961 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.707647 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.707633 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.707712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.707660 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.707712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.707640 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.707712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.707698 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.707712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.707674 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.707822 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.707715 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.708724 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.708705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.708724 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.708728 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 19:15:23.709403 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.709386 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientMemory" Aug 04 19:15:23.709490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.709420 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 19:15:23.709490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.709431 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeHasSufficientPID" Aug 04 19:15:23.734319 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.734292 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-62.ec2.internal\" not found" node="ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.738770 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.738753 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-62.ec2.internal\" not found" node="ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.769620 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.769594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28866fd69f4b6b3789ec934331dbf568-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal\" (UID: \"28866fd69f4b6b3789ec934331dbf568\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.769620 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.769624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28866fd69f4b6b3789ec934331dbf568-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal\" (UID: \"28866fd69f4b6b3789ec934331dbf568\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.769782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.769647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76dbb7836f721215f5525cdef4bf64f8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-62.ec2.internal\" (UID: \"76dbb7836f721215f5525cdef4bf64f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.770651 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.770627 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:23.870770 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.870733 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:23.870949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.870795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28866fd69f4b6b3789ec934331dbf568-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal\" (UID: \"28866fd69f4b6b3789ec934331dbf568\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.870949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.870821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28866fd69f4b6b3789ec934331dbf568-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal\" (UID: \"28866fd69f4b6b3789ec934331dbf568\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.870949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.870839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76dbb7836f721215f5525cdef4bf64f8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-62.ec2.internal\" (UID: \"76dbb7836f721215f5525cdef4bf64f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.870949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.870879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76dbb7836f721215f5525cdef4bf64f8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-62.ec2.internal\" (UID: \"76dbb7836f721215f5525cdef4bf64f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.870949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.870906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28866fd69f4b6b3789ec934331dbf568-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal\" (UID: \"28866fd69f4b6b3789ec934331dbf568\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.870949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:23.870905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28866fd69f4b6b3789ec934331dbf568-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal\" (UID: \"28866fd69f4b6b3789ec934331dbf568\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:23.971640 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:23.971552 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.036729 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.036699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:24.042821 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.042802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" Aug 04 19:15:24.072674 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.072635 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.173067 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.173038 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.273571 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.273511 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.373933 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.373892 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.407088 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.407041 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 19:15:24.407599 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.407228 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 19:15:24.474422 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.474383 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.498191 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.498134 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 19:10:23 +0000 UTC" deadline="2028-01-02 22:26:26.510610156 +0000 UTC" Aug 04 19:15:24.498191 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.498184 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12387h11m2.012429569s" Aug 04 19:15:24.554786 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:24.554756 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28866fd69f4b6b3789ec934331dbf568.slice/crio-65477f8940429d9a780fd455f84443fb01fb114800fd0560f311e6eed7f7fcaa WatchSource:0}: Error finding container 65477f8940429d9a780fd455f84443fb01fb114800fd0560f311e6eed7f7fcaa: Status 404 returned error can't find the container with id 65477f8940429d9a780fd455f84443fb01fb114800fd0560f311e6eed7f7fcaa Aug 04 19:15:24.555437 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:24.555404 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dbb7836f721215f5525cdef4bf64f8.slice/crio-eb9c46cafa87f06e0b42340cbc47de38338bd6ef49c081411f5e7586ae0fddf2 WatchSource:0}: Error finding container eb9c46cafa87f06e0b42340cbc47de38338bd6ef49c081411f5e7586ae0fddf2: Status 404 returned error can't find the container with id eb9c46cafa87f06e0b42340cbc47de38338bd6ef49c081411f5e7586ae0fddf2 Aug 04 19:15:24.559895 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.559873 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:15:24.572490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.572468 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:24.575020 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.575002 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.607677 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.607615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" event={"ID":"76dbb7836f721215f5525cdef4bf64f8","Type":"ContainerStarted","Data":"eb9c46cafa87f06e0b42340cbc47de38338bd6ef49c081411f5e7586ae0fddf2"} Aug 04 19:15:24.608713 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.608690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" event={"ID":"28866fd69f4b6b3789ec934331dbf568","Type":"ContainerStarted","Data":"65477f8940429d9a780fd455f84443fb01fb114800fd0560f311e6eed7f7fcaa"} Aug 04 19:15:24.676078 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.676050 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.776508 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:24.776472 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-62.ec2.internal\" not found" Aug 04 19:15:24.804521 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.804440 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:24.869746 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.869715 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:24.967805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.967776 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" Aug 04 19:15:24.984916 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.984875 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 19:15:24.986066 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.986044 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" Aug 04 19:15:24.994547 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:24.994515 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 19:15:25.355048 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.355019 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 19:15:25.449494 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.449457 2579 apiserver.go:52] "Watching apiserver" Aug 04 19:15:25.455416 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.455362 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 19:15:25.456607 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.456582 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qnd99","kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq","openshift-dns/node-resolver-b6mw2","openshift-network-diagnostics/network-check-target-mm924","openshift-network-operator/iptables-alerter-s8jwr","openshift-ovn-kubernetes/ovnkube-node-99kml","kube-system/global-pull-secret-syncer-glb96","kube-system/konnectivity-agent-9f99v","openshift-cluster-node-tuning-operator/tuned-pdgjz","openshift-image-registry/node-ca-4sdzl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-2sh5f","openshift-multus/multus-xcnvp"] Aug 04 19:15:25.458036 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.458016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.459163 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.459144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.460290 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.460270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.461072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.461036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.462308 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.461687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.462308 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.461802 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.462308 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.461920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zn78z\"" Aug 04 19:15:25.462834 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.462724 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.462921 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.462836 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 19:15:25.463803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.463063 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 19:15:25.463803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.463097 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 19:15:25.463803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.463392 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 19:15:25.463803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.463442 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 19:15:25.464750 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.464721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:25.464856 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.464832 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.464924 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.464864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:25.464983 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.464929 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 19:15:25.464983 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.464973 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.465076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465004 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrqkv\"" Aug 04 19:15:25.465076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lc8tl\"" Aug 04 19:15:25.465184 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465165 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.465235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465209 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 19:15:25.465385 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465169 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ntkm8\"" Aug 04 19:15:25.465385 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465283 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.465545 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465527 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 19:15:25.465645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.465614 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.468393 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.468376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.468814 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.468797 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 19:15:25.469574 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.469547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.470592 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.470573 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.471003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.470616 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.471003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.470748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7kpm\"" Aug 04 19:15:25.471003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.470761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:25.471003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.470645 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 19:15:25.471003 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.470825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:25.471647 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.471630 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 19:15:25.471828 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.471811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 19:15:25.471960 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.471941 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4s5dp\"" Aug 04 19:15:25.472315 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.472301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.473408 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.473386 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.474275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.474251 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 19:15:25.474554 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.474538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.474894 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.474724 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-swlrh\"" Aug 04 19:15:25.475915 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.475903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.475984 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.475959 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:25.476134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.476117 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.477055 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.477036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmh2g\"" Aug 04 19:15:25.477259 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.477238 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.477355 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.477255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 19:15:25.477419 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.477401 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lwvdr\"" Aug 04 19:15:25.477468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.477439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 19:15:25.478075 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.477439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 19:15:25.479930 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.479908 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 19:15:25.481030 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-var-lib-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-socket-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.481107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.481234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-cni-bin\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-system-cni-dir\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.481358 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.481358 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.481464 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-systemd-units\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481464 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-log-socket\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481464 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-slash\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481464 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-run-netns\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481464 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-etc-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-node-log\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-ovnkube-config\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6a265ca-059b-46f3-b9d9-a0589199d67c-konnectivity-ca\") pod \"konnectivity-agent-9f99v\" (UID: \"a6a265ca-059b-46f3-b9d9-a0589199d67c\") " pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258893ce-f863-4989-bfb6-2076f423bad4-tmp-dir\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6wgf\" (UniqueName: \"kubernetes.io/projected/258893ce-f863-4989-bfb6-2076f423bad4-kube-api-access-q6wgf\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-cni-netd\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.481681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66dbcaea-5165-425d-9573-caddd6f40f06-ovn-node-metrics-cert\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-ovnkube-script-lib\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-sys-fs\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnc6r\" (UniqueName: \"kubernetes.io/projected/de87a6e0-0fec-4b8c-a452-ddb86083c823-kube-api-access-xnc6r\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-cnibin\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lcv\" (UniqueName: \"kubernetes.io/projected/1e75a8e6-9463-4849-adaf-7b2cadc09529-kube-api-access-g4lcv\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-kubelet\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nws7m\" (UniqueName: \"kubernetes.io/projected/66dbcaea-5165-425d-9573-caddd6f40f06-kube-api-access-nws7m\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-device-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6a265ca-059b-46f3-b9d9-a0589199d67c-agent-certs\") pod \"konnectivity-agent-9f99v\" (UID: \"a6a265ca-059b-46f3-b9d9-a0589199d67c\") " pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.482016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.481985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-systemd\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-env-overrides\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-registration-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-etc-selinux\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de87a6e0-0fec-4b8c-a452-ddb86083c823-iptables-alerter-script\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-os-release\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-ovn\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wj4f\" (UniqueName: \"kubernetes.io/projected/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-kube-api-access-5wj4f\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de87a6e0-0fec-4b8c-a452-ddb86083c823-host-slash\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.482730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.482286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/258893ce-f863-4989-bfb6-2076f423bad4-hosts-file\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.501153 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.501126 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vb6zb" Aug 04 19:15:25.508878 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.508855 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vb6zb" Aug 04 19:15:25.569472 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.569445 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 19:15:25.583324 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/209159ed-1897-4041-98cc-a925e9fe5306-multus-daemon-config\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.583324 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysconfig\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.583553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-var-lib-kubelet\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.583553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-systemd-units\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-systemd-units\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-log-socket\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-system-cni-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.583553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-log-socket\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-cnibin\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-run\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-slash\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-slash\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-run-netns\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-node-log\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6a265ca-059b-46f3-b9d9-a0589199d67c-konnectivity-ca\") pod \"konnectivity-agent-9f99v\" (UID: \"a6a265ca-059b-46f3-b9d9-a0589199d67c\") " pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-run-netns\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6wgf\" (UniqueName: \"kubernetes.io/projected/258893ce-f863-4989-bfb6-2076f423bad4-kube-api-access-q6wgf\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.583785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-cni-netd\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-node-log\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-cni-netd\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-sys-fs\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnc6r\" (UniqueName: \"kubernetes.io/projected/de87a6e0-0fec-4b8c-a452-ddb86083c823-kube-api-access-xnc6r\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-cni-multus\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.583953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-sys-fs\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-kubelet\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-kubelet\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-kubelet\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-device-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6a265ca-059b-46f3-b9d9-a0589199d67c-agent-certs\") pod \"konnectivity-agent-9f99v\" (UID: \"a6a265ca-059b-46f3-b9d9-a0589199d67c\") " pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-multus-certs\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.584310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-systemd\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-systemd\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-env-overrides\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584446 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-k8s-cni-cncf-io\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-device-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysctl-d\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e71b5a85-d145-400f-9157-cfafbdbc0c4b-tmp\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584632 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de87a6e0-0fec-4b8c-a452-ddb86083c823-host-slash\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/258893ce-f863-4989-bfb6-2076f423bad4-hosts-file\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-kubernetes\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc25m\" (UniqueName: \"kubernetes.io/projected/e71b5a85-d145-400f-9157-cfafbdbc0c4b-kube-api-access-tc25m\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-var-lib-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/258893ce-f863-4989-bfb6-2076f423bad4-hosts-file\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-etc-kubernetes\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.585081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-env-overrides\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-var-lib-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-lib-modules\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.584933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de87a6e0-0fec-4b8c-a452-ddb86083c823-host-slash\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-system-cni-dir\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-system-cni-dir\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585063 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6a265ca-059b-46f3-b9d9-a0589199d67c-konnectivity-ca\") pod \"konnectivity-agent-9f99v\" (UID: \"a6a265ca-059b-46f3-b9d9-a0589199d67c\") " pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-socket-dir-parent\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-modprobe-d\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-etc-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-ovnkube-config\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258893ce-f863-4989-bfb6-2076f423bad4-tmp-dir\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2mm\" (UniqueName: \"kubernetes.io/projected/2a8860b7-1f1b-4269-b469-cac0bc30c406-kube-api-access-kj2mm\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-conf-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-etc-openvswitch\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4c10f094-a78b-4733-b541-57206eefdc6f-kubelet-config\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66dbcaea-5165-425d-9573-caddd6f40f06-ovn-node-metrics-cert\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.585872 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-ovnkube-script-lib\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-cnibin\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lcv\" (UniqueName: \"kubernetes.io/projected/1e75a8e6-9463-4849-adaf-7b2cadc09529-kube-api-access-g4lcv\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-netns\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-hostroot\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585562 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-cnibin\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysctl-conf\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-tuned\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nws7m\" (UniqueName: \"kubernetes.io/projected/66dbcaea-5165-425d-9573-caddd6f40f06-kube-api-access-nws7m\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258893ce-f863-4989-bfb6-2076f423bad4-tmp-dir\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-sys\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-host\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-host\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-serviceca\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.586682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-ovnkube-config\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4vz\" (UniqueName: \"kubernetes.io/projected/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-kube-api-access-9z4vz\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-registration-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/66dbcaea-5165-425d-9573-caddd6f40f06-ovnkube-script-lib\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.585992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-etc-selinux\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de87a6e0-0fec-4b8c-a452-ddb86083c823-iptables-alerter-script\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-registration-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-os-release\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586074 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-etc-selinux\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/209159ed-1897-4041-98cc-a925e9fe5306-cni-binary-copy\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-ovn\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e75a8e6-9463-4849-adaf-7b2cadc09529-os-release\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wj4f\" (UniqueName: \"kubernetes.io/projected/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-kube-api-access-5wj4f\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-cni-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-run-ovn\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-os-release\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qdr\" (UniqueName: \"kubernetes.io/projected/209159ed-1897-4041-98cc-a925e9fe5306-kube-api-access-69qdr\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.587491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-systemd\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4c10f094-a78b-4733-b541-57206eefdc6f-dbus\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-socket-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-cni-bin\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de87a6e0-0fec-4b8c-a452-ddb86083c823-iptables-alerter-script\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-cni-bin\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66dbcaea-5165-425d-9573-caddd6f40f06-host-cni-bin\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.586596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-socket-dir\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.587086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.587111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-cni-binary-copy\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.588168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.587629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e75a8e6-9463-4849-adaf-7b2cadc09529-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.588930 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.588512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6a265ca-059b-46f3-b9d9-a0589199d67c-agent-certs\") pod \"konnectivity-agent-9f99v\" (UID: \"a6a265ca-059b-46f3-b9d9-a0589199d67c\") " pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.588930 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.588771 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/66dbcaea-5165-425d-9573-caddd6f40f06-ovn-node-metrics-cert\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.592839 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.592566 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:25.592839 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.592589 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:25.592839 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.592602 2579 projected.go:196] Error preparing data for projected volume kube-api-access-24dp2 for pod openshift-network-diagnostics/network-check-target-mm924: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:25.592839 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.592657 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2 podName:92e0a43b-f828-47fc-86af-81f2dbd29877 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:26.09263862 +0000 UTC m=+3.033435361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-24dp2" (UniqueName: "kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2") pod "network-check-target-mm924" (UID: "92e0a43b-f828-47fc-86af-81f2dbd29877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:25.594022 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.593962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nws7m\" (UniqueName: \"kubernetes.io/projected/66dbcaea-5165-425d-9573-caddd6f40f06-kube-api-access-nws7m\") pod \"ovnkube-node-99kml\" (UID: \"66dbcaea-5165-425d-9573-caddd6f40f06\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.594321 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.594300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lcv\" (UniqueName: \"kubernetes.io/projected/1e75a8e6-9463-4849-adaf-7b2cadc09529-kube-api-access-g4lcv\") pod \"multus-additional-cni-plugins-2sh5f\" (UID: \"1e75a8e6-9463-4849-adaf-7b2cadc09529\") " pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.594791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.594773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnc6r\" (UniqueName: \"kubernetes.io/projected/de87a6e0-0fec-4b8c-a452-ddb86083c823-kube-api-access-xnc6r\") pod \"iptables-alerter-s8jwr\" (UID: \"de87a6e0-0fec-4b8c-a452-ddb86083c823\") " pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.595161 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.595143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wj4f\" (UniqueName: \"kubernetes.io/projected/b340ed23-8473-48d2-be0a-53ef9b9ddb8c-kube-api-access-5wj4f\") pod \"aws-ebs-csi-driver-node-789xq\" (UID: \"b340ed23-8473-48d2-be0a-53ef9b9ddb8c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.595969 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.595951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6wgf\" (UniqueName: \"kubernetes.io/projected/258893ce-f863-4989-bfb6-2076f423bad4-kube-api-access-q6wgf\") pod \"node-resolver-b6mw2\" (UID: \"258893ce-f863-4989-bfb6-2076f423bad4\") " pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.687942 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:25.687942 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-cni-multus\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.687942 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-kubelet\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.687942 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-multus-certs\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-k8s-cni-cncf-io\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysctl-d\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.687995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-cni-multus\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-kubelet\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e71b5a85-d145-400f-9157-cfafbdbc0c4b-tmp\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.688054 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-kubernetes\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-multus-certs\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc25m\" (UniqueName: \"kubernetes.io/projected/e71b5a85-d145-400f-9157-cfafbdbc0c4b-kube-api-access-tc25m\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-k8s-cni-cncf-io\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.688139 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:26.188117369 +0000 UTC m=+3.128914107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-kubernetes\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysctl-d\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-etc-kubernetes\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-lib-modules\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.688262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-socket-dir-parent\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-etc-kubernetes\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-modprobe-d\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2mm\" (UniqueName: \"kubernetes.io/projected/2a8860b7-1f1b-4269-b469-cac0bc30c406-kube-api-access-kj2mm\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-socket-dir-parent\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-lib-modules\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-conf-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4c10f094-a78b-4733-b541-57206eefdc6f-kubelet-config\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-modprobe-d\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-netns\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-run-netns\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-hostroot\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-conf-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysctl-conf\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-hostroot\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-tuned\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4c10f094-a78b-4733-b541-57206eefdc6f-kubelet-config\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-sys\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689020 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-host\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-sys\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-host\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysctl-conf\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-serviceca\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-host\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-host\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4vz\" (UniqueName: \"kubernetes.io/projected/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-kube-api-access-9z4vz\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/209159ed-1897-4041-98cc-a925e9fe5306-cni-binary-copy\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-cni-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-os-release\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qdr\" (UniqueName: \"kubernetes.io/projected/209159ed-1897-4041-98cc-a925e9fe5306-kube-api-access-69qdr\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-systemd\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4c10f094-a78b-4733-b541-57206eefdc6f-dbus\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-cni-bin\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/209159ed-1897-4041-98cc-a925e9fe5306-multus-daemon-config\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-multus-cni-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.689782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.688987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysconfig\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-var-lib-kubelet\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-system-cni-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-cnibin\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-run\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4c10f094-a78b-4733-b541-57206eefdc6f-dbus\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-os-release\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-system-cni-dir\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.689056 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-serviceca\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689205 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-sysconfig\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-host-var-lib-cni-bin\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-systemd\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-run\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:25.689300 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:15:26.189281711 +0000 UTC m=+3.130078440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/209159ed-1897-4041-98cc-a925e9fe5306-cnibin\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e71b5a85-d145-400f-9157-cfafbdbc0c4b-var-lib-kubelet\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/209159ed-1897-4041-98cc-a925e9fe5306-cni-binary-copy\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.690525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.689591 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/209159ed-1897-4041-98cc-a925e9fe5306-multus-daemon-config\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.691369 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.690991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e71b5a85-d145-400f-9157-cfafbdbc0c4b-tmp\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.691369 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.691163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e71b5a85-d145-400f-9157-cfafbdbc0c4b-etc-tuned\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.696956 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.696925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qdr\" (UniqueName: \"kubernetes.io/projected/209159ed-1897-4041-98cc-a925e9fe5306-kube-api-access-69qdr\") pod \"multus-xcnvp\" (UID: \"209159ed-1897-4041-98cc-a925e9fe5306\") " pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.697280 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.697248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4vz\" (UniqueName: \"kubernetes.io/projected/d27cb5fd-9a5d-4afa-b3c1-177a83080e08-kube-api-access-9z4vz\") pod \"node-ca-4sdzl\" (UID: \"d27cb5fd-9a5d-4afa-b3c1-177a83080e08\") " pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:25.697550 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.697532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2mm\" (UniqueName: \"kubernetes.io/projected/2a8860b7-1f1b-4269-b469-cac0bc30c406-kube-api-access-kj2mm\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:25.697671 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.697651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc25m\" (UniqueName: \"kubernetes.io/projected/e71b5a85-d145-400f-9157-cfafbdbc0c4b-kube-api-access-tc25m\") pod \"tuned-pdgjz\" (UID: \"e71b5a85-d145-400f-9157-cfafbdbc0c4b\") " pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.771390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.771302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:25.777996 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.777962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" Aug 04 19:15:25.785576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.785552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" Aug 04 19:15:25.791241 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.791217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b6mw2" Aug 04 19:15:25.798694 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.798673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s8jwr" Aug 04 19:15:25.805190 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.805172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:25.811523 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.811500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xcnvp" Aug 04 19:15:25.818928 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.818905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" Aug 04 19:15:25.825367 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:25.825344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4sdzl" Aug 04 19:15:26.192826 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.192798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:26.193013 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.192856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:26.193013 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.192890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:26.193013 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.192974 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:26.193013 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193007 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:26.193204 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193019 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:26.193204 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193038 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:26.193204 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193050 2579 projected.go:196] Error preparing data for projected volume kube-api-access-24dp2 for pod openshift-network-diagnostics/network-check-target-mm924: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:26.193204 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193053 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:27.193033541 +0000 UTC m=+4.133830282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:26.193204 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193082 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:15:27.19306315 +0000 UTC m=+4.133859878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:26.193204 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.193097 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2 podName:92e0a43b-f828-47fc-86af-81f2dbd29877 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:27.193089004 +0000 UTC m=+4.133885730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-24dp2" (UniqueName: "kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2") pod "network-check-target-mm924" (UID: "92e0a43b-f828-47fc-86af-81f2dbd29877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:26.424763 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.424739 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e75a8e6_9463_4849_adaf_7b2cadc09529.slice/crio-d6e53e4fbad90ff80f60975841fdd602e06ba7696b7cfe5e694451f10910dd44 WatchSource:0}: Error finding container d6e53e4fbad90ff80f60975841fdd602e06ba7696b7cfe5e694451f10910dd44: Status 404 returned error can't find the container with id d6e53e4fbad90ff80f60975841fdd602e06ba7696b7cfe5e694451f10910dd44 Aug 04 19:15:26.426042 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.426020 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb340ed23_8473_48d2_be0a_53ef9b9ddb8c.slice/crio-8ba69a58c612e49f5b570ca58e2fb1f041e5592e0ea7ee2e3d8ee100229c4929 WatchSource:0}: Error finding container 8ba69a58c612e49f5b570ca58e2fb1f041e5592e0ea7ee2e3d8ee100229c4929: Status 404 returned error can't find the container with id 8ba69a58c612e49f5b570ca58e2fb1f041e5592e0ea7ee2e3d8ee100229c4929 Aug 04 19:15:26.427234 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.427159 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258893ce_f863_4989_bfb6_2076f423bad4.slice/crio-ee8ae0aa5dd76a70db29070966060177ee356c9b8d3fbfd3595a1b4fa09205f6 WatchSource:0}: Error finding container ee8ae0aa5dd76a70db29070966060177ee356c9b8d3fbfd3595a1b4fa09205f6: Status 404 returned error can't find the container with id ee8ae0aa5dd76a70db29070966060177ee356c9b8d3fbfd3595a1b4fa09205f6 Aug 04 19:15:26.428947 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.428918 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27cb5fd_9a5d_4afa_b3c1_177a83080e08.slice/crio-21224ab20634b809aaff2493b15048a09a9c11160b6cb10ffcf9b94e7959af8f WatchSource:0}: Error finding container 21224ab20634b809aaff2493b15048a09a9c11160b6cb10ffcf9b94e7959af8f: Status 404 returned error can't find the container with id 21224ab20634b809aaff2493b15048a09a9c11160b6cb10ffcf9b94e7959af8f Aug 04 19:15:26.431706 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.431683 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209159ed_1897_4041_98cc_a925e9fe5306.slice/crio-ef84c7eab2803c3740b061a7f5db95a84b3a8464bd88d0cc7c67d7832c8eb896 WatchSource:0}: Error finding container ef84c7eab2803c3740b061a7f5db95a84b3a8464bd88d0cc7c67d7832c8eb896: Status 404 returned error can't find the container with id ef84c7eab2803c3740b061a7f5db95a84b3a8464bd88d0cc7c67d7832c8eb896 Aug 04 19:15:26.433047 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.433025 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71b5a85_d145_400f_9157_cfafbdbc0c4b.slice/crio-bafc7f7e2a7b47f66fac89e289f5076febb3206435b12163374f039d34e21ce0 WatchSource:0}: Error finding container bafc7f7e2a7b47f66fac89e289f5076febb3206435b12163374f039d34e21ce0: Status 404 returned error can't find the container with id bafc7f7e2a7b47f66fac89e289f5076febb3206435b12163374f039d34e21ce0 Aug 04 19:15:26.434492 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.434471 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dbcaea_5165_425d_9573_caddd6f40f06.slice/crio-1852c0d2b50b028b9c54c78172ffc1db7d42c3476f78ace87508ae6de0b06958 WatchSource:0}: Error finding container 1852c0d2b50b028b9c54c78172ffc1db7d42c3476f78ace87508ae6de0b06958: Status 404 returned error can't find the container with id 1852c0d2b50b028b9c54c78172ffc1db7d42c3476f78ace87508ae6de0b06958 Aug 04 19:15:26.436414 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.436380 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde87a6e0_0fec_4b8c_a452_ddb86083c823.slice/crio-2b3de06c6b94386120a92a384f8fc8dac0f4fc53bf322c51cb91536f5576df76 WatchSource:0}: Error finding container 2b3de06c6b94386120a92a384f8fc8dac0f4fc53bf322c51cb91536f5576df76: Status 404 returned error can't find the container with id 2b3de06c6b94386120a92a384f8fc8dac0f4fc53bf322c51cb91536f5576df76 Aug 04 19:15:26.437286 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:26.437200 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a265ca_059b_46f3_b9d9_a0589199d67c.slice/crio-59ff5b6146faecff500f88f1cbb3048eb47fc7af4ccc3f0bc94513dcba71e79f WatchSource:0}: Error finding container 59ff5b6146faecff500f88f1cbb3048eb47fc7af4ccc3f0bc94513dcba71e79f: Status 404 returned error can't find the container with id 59ff5b6146faecff500f88f1cbb3048eb47fc7af4ccc3f0bc94513dcba71e79f Aug 04 19:15:26.510996 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.510946 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 19:10:25 +0000 UTC" deadline="2028-03-07 09:30:07.967315922 +0000 UTC" Aug 04 19:15:26.510996 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.510995 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13934h14m41.456324507s" Aug 04 19:15:26.604130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.603898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:26.604265 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.603928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:26.604265 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.604223 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:26.604362 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:26.604260 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:26.613456 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.613429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4sdzl" event={"ID":"d27cb5fd-9a5d-4afa-b3c1-177a83080e08","Type":"ContainerStarted","Data":"21224ab20634b809aaff2493b15048a09a9c11160b6cb10ffcf9b94e7959af8f"} Aug 04 19:15:26.614661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.614627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerStarted","Data":"d6e53e4fbad90ff80f60975841fdd602e06ba7696b7cfe5e694451f10910dd44"} Aug 04 19:15:26.616100 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.616073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s8jwr" event={"ID":"de87a6e0-0fec-4b8c-a452-ddb86083c823","Type":"ContainerStarted","Data":"2b3de06c6b94386120a92a384f8fc8dac0f4fc53bf322c51cb91536f5576df76"} Aug 04 19:15:26.618693 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.618668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"1852c0d2b50b028b9c54c78172ffc1db7d42c3476f78ace87508ae6de0b06958"} Aug 04 19:15:26.621262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.621242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xcnvp" event={"ID":"209159ed-1897-4041-98cc-a925e9fe5306","Type":"ContainerStarted","Data":"ef84c7eab2803c3740b061a7f5db95a84b3a8464bd88d0cc7c67d7832c8eb896"} Aug 04 19:15:26.622192 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.622173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b6mw2" event={"ID":"258893ce-f863-4989-bfb6-2076f423bad4","Type":"ContainerStarted","Data":"ee8ae0aa5dd76a70db29070966060177ee356c9b8d3fbfd3595a1b4fa09205f6"} Aug 04 19:15:26.624133 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.624111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" event={"ID":"b340ed23-8473-48d2-be0a-53ef9b9ddb8c","Type":"ContainerStarted","Data":"8ba69a58c612e49f5b570ca58e2fb1f041e5592e0ea7ee2e3d8ee100229c4929"} Aug 04 19:15:26.626355 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.626313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" event={"ID":"76dbb7836f721215f5525cdef4bf64f8","Type":"ContainerStarted","Data":"7b1ce625e583a238f83d97972b534956b756ef37835ac16617a891c7bfc7997f"} Aug 04 19:15:26.630040 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.630012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9f99v" event={"ID":"a6a265ca-059b-46f3-b9d9-a0589199d67c","Type":"ContainerStarted","Data":"59ff5b6146faecff500f88f1cbb3048eb47fc7af4ccc3f0bc94513dcba71e79f"} Aug 04 19:15:26.631008 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.630988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" event={"ID":"e71b5a85-d145-400f-9157-cfafbdbc0c4b","Type":"ContainerStarted","Data":"bafc7f7e2a7b47f66fac89e289f5076febb3206435b12163374f039d34e21ce0"} Aug 04 19:15:26.639670 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:26.639613 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-62.ec2.internal" podStartSLOduration=2.639596574 podStartE2EDuration="2.639596574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:26.639535898 +0000 UTC m=+3.580332640" watchObservedRunningTime="2026-08-04 19:15:26.639596574 +0000 UTC m=+3.580393317" Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.201905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.201964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.202007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202137 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202143 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202155 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202166 2579 projected.go:196] Error preparing data for projected volume kube-api-access-24dp2 for pod openshift-network-diagnostics/network-check-target-mm924: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202201 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:15:29.202183272 +0000 UTC m=+6.142980007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202218 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2 podName:92e0a43b-f828-47fc-86af-81f2dbd29877 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:29.202209889 +0000 UTC m=+6.143006611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-24dp2" (UniqueName: "kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2") pod "network-check-target-mm924" (UID: "92e0a43b-f828-47fc-86af-81f2dbd29877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202233 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:27.202291 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.202259 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:29.202249381 +0000 UTC m=+6.143046104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:27.511700 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.511573 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 19:10:25 +0000 UTC" deadline="2028-03-12 04:30:59.089236095 +0000 UTC" Aug 04 19:15:27.511700 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.511632 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14049h15m31.577608052s" Aug 04 19:15:27.607109 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.606557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:27.607109 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:27.606699 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:27.639811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.638506 2579 generic.go:356] "Generic (PLEG): container finished" podID="28866fd69f4b6b3789ec934331dbf568" containerID="d9b8270c4a96eddf6f68fc1545fff7e4dc46654e78b621a06776a1135bc8927e" exitCode=0 Aug 04 19:15:27.639811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:27.639524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" event={"ID":"28866fd69f4b6b3789ec934331dbf568","Type":"ContainerDied","Data":"d9b8270c4a96eddf6f68fc1545fff7e4dc46654e78b621a06776a1135bc8927e"} Aug 04 19:15:28.604107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:28.604069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:28.604520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:28.604069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:28.604520 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:28.604209 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:28.604520 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:28.604247 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:28.643479 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:28.643442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" event={"ID":"28866fd69f4b6b3789ec934331dbf568","Type":"ContainerStarted","Data":"34d8252249e94aad425cbef9117be489201f30e4aed0ebcc62b75f84b032359a"} Aug 04 19:15:28.660314 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:28.660200 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-62.ec2.internal" podStartSLOduration=4.660182336 podStartE2EDuration="4.660182336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:28.659598466 +0000 UTC m=+5.600395212" watchObservedRunningTime="2026-08-04 19:15:28.660182336 +0000 UTC m=+5.600979081" Aug 04 19:15:29.220220 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:29.220181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:29.220437 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:29.220238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:29.220437 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:29.220280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:29.220437 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220375 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:29.220437 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220395 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:29.220437 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220403 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:29.220638 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220446 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:33.220432935 +0000 UTC m=+10.161229659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:29.220638 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220395 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:29.220638 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220459 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:15:33.220452722 +0000 UTC m=+10.161249446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:29.220638 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220466 2579 projected.go:196] Error preparing data for projected volume kube-api-access-24dp2 for pod openshift-network-diagnostics/network-check-target-mm924: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:29.220638 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.220509 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2 podName:92e0a43b-f828-47fc-86af-81f2dbd29877 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:33.220489898 +0000 UTC m=+10.161286637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-24dp2" (UniqueName: "kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2") pod "network-check-target-mm924" (UID: "92e0a43b-f828-47fc-86af-81f2dbd29877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:29.604569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:29.604488 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:29.604986 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:29.604637 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:30.603935 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:30.603901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:30.604125 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:30.604041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:30.604125 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:30.604112 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:30.604228 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:30.604198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:31.606625 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:31.606087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:31.606625 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:31.606242 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:32.604340 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:32.604228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:32.604520 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:32.604385 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:32.604520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:32.604481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:32.604633 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:32.604586 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:33.255539 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:33.255455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:33.255539 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:33.255509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:33.255549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255663 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255694 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255707 2579 projected.go:196] Error preparing data for projected volume kube-api-access-24dp2 for pod openshift-network-diagnostics/network-check-target-mm924: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255720 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255668 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255767 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2 podName:92e0a43b-f828-47fc-86af-81f2dbd29877 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:41.255750189 +0000 UTC m=+18.196546915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-24dp2" (UniqueName: "kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2") pod "network-check-target-mm924" (UID: "92e0a43b-f828-47fc-86af-81f2dbd29877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255792 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:41.255775969 +0000 UTC m=+18.196572697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:33.256028 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.255810 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:15:41.255801238 +0000 UTC m=+18.196597961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:33.605204 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:33.605163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:33.605392 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:33.605313 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:34.604088 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:34.604054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:34.604582 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:34.604178 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:34.604654 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:34.604588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:34.604762 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:34.604710 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:35.604957 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:35.604793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:35.604957 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:35.604940 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:36.605287 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.604648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:36.605287 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:36.604775 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:36.605287 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.604848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:36.605287 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:36.604914 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:36.669242 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.669170 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" event={"ID":"b340ed23-8473-48d2-be0a-53ef9b9ddb8c","Type":"ContainerStarted","Data":"bd1bbf057799f99993f4df3db4f6f64acf7f548e2135c6c9e0aa8a5bfc9ca928"} Aug 04 19:15:36.674228 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.674161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" event={"ID":"e71b5a85-d145-400f-9157-cfafbdbc0c4b","Type":"ContainerStarted","Data":"4db470a5a5cf8a06597203e6ce47f70ce3806e95ec20529c56f691e6b4001a98"} Aug 04 19:15:36.676448 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.676305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4sdzl" event={"ID":"d27cb5fd-9a5d-4afa-b3c1-177a83080e08","Type":"ContainerStarted","Data":"1fea4554d4d67753aa18b38a6c3eb71e51b2335813725505d33594dfd8fa70f9"} Aug 04 19:15:36.679092 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.678833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerStarted","Data":"5c6e8cceae26757081230bfdc29bb2c13ebbf8f8369b324723b3038ed1085645"} Aug 04 19:15:36.690914 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.690856 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pdgjz" podStartSLOduration=3.706341119 podStartE2EDuration="13.690838805s" totalImagesPullingTime="9.984497686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:36.690759231 +0000 UTC m=+13.631555979" watchObservedRunningTime="2026-08-04 19:15:36.690838805 +0000 UTC m=+13.631635552" Aug 04 19:15:36.703771 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:36.703704 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4sdzl" podStartSLOduration=4.105833454 podStartE2EDuration="13.703683699s" totalImagesPullingTime="9.597850245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:36.703507941 +0000 UTC m=+13.644304686" watchObservedRunningTime="2026-08-04 19:15:36.703683699 +0000 UTC m=+13.644480446" Aug 04 19:15:37.604821 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:37.604562 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:37.605011 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:37.604925 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:37.683383 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:37.683343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b6mw2" event={"ID":"258893ce-f863-4989-bfb6-2076f423bad4","Type":"ContainerStarted","Data":"1e498f695e2c062f0d7eb2bf8387cd15fa4fefbd75083dd54ba387ed019032ae"} Aug 04 19:15:37.685386 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:37.685351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9f99v" event={"ID":"a6a265ca-059b-46f3-b9d9-a0589199d67c","Type":"ContainerStarted","Data":"873f9ff0adc6529b8b76a4bd6148a262629a48fd98d9b71a6f28659291ce5b82"} Aug 04 19:15:37.700744 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:37.700682 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b6mw2" podStartSLOduration=5.102218293 podStartE2EDuration="14.700658081s" totalImagesPullingTime="9.598439788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:37.70056179 +0000 UTC m=+14.641358550" watchObservedRunningTime="2026-08-04 19:15:37.700658081 +0000 UTC m=+14.641454830" Aug 04 19:15:37.740514 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:37.740457 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9f99v" podStartSLOduration=4.776524415 podStartE2EDuration="14.740439475s" totalImagesPullingTime="9.96391506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:37.738756155 +0000 UTC m=+14.679552902" watchObservedRunningTime="2026-08-04 19:15:37.740439475 +0000 UTC m=+14.681236304" Aug 04 19:15:38.604255 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:38.604219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:38.604451 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:38.604224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:38.604451 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:38.604367 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:38.604564 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:38.604444 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:38.688925 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:38.688880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s8jwr" event={"ID":"de87a6e0-0fec-4b8c-a452-ddb86083c823","Type":"ContainerStarted","Data":"0ba365a8c33992a4ab14e2ec534a279584f8d58c33592d0d3b5f68ba77d1e1b9"} Aug 04 19:15:38.704520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:38.704468 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s8jwr" podStartSLOduration=6.114380911 podStartE2EDuration="15.704451181s" totalImagesPullingTime="9.59007027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:38.704026528 +0000 UTC m=+15.644823274" watchObservedRunningTime="2026-08-04 19:15:38.704451181 +0000 UTC m=+15.645247924" Aug 04 19:15:39.604789 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:39.604744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:39.605008 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:39.604876 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:40.604232 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:40.604184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:40.604915 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:40.604184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:40.604915 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:40.604348 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:40.604915 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:40.604452 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:41.323430 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:41.323391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:41.323672 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:41.323441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:41.323672 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323612 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 19:15:41.323672 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323646 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 19:15:41.323672 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323660 2579 projected.go:196] Error preparing data for projected volume kube-api-access-24dp2 for pod openshift-network-diagnostics/network-check-target-mm924: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:41.323829 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323703 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:41.323829 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323724 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2 podName:92e0a43b-f828-47fc-86af-81f2dbd29877 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.323705591 +0000 UTC m=+34.264502330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-24dp2" (UniqueName: "kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2") pod "network-check-target-mm924" (UID: "92e0a43b-f828-47fc-86af-81f2dbd29877") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 19:15:41.323829 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323760 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.323744329 +0000 UTC m=+34.264541078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:41.323829 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323816 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:41.323992 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.323848 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.323837471 +0000 UTC m=+34.264634208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:41.323992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:41.323611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:41.604051 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:41.603958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:41.604202 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:41.604125 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:41.697562 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:41.697530 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e75a8e6-9463-4849-adaf-7b2cadc09529" containerID="5c6e8cceae26757081230bfdc29bb2c13ebbf8f8369b324723b3038ed1085645" exitCode=0 Aug 04 19:15:41.698100 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:41.697575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerDied","Data":"5c6e8cceae26757081230bfdc29bb2c13ebbf8f8369b324723b3038ed1085645"} Aug 04 19:15:42.359512 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:42.359457 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:42.360308 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:42.360281 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:42.603939 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:42.603903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:42.603939 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:42.603944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:42.604163 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:42.604045 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:42.604254 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:42.604221 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:43.605215 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:43.605176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:43.605832 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:43.605299 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:44.203005 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:44.202969 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:44.203195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:44.203094 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 19:15:44.203557 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:44.203532 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9f99v" Aug 04 19:15:44.604774 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:44.604737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:44.604928 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:44.604849 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:44.604928 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:44.604896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:44.605013 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:44.604996 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:45.294709 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.294460 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 19:15:45.544011 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.543893 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T19:15:45.294702953Z","UUID":"40b65eb8-33a5-403a-b2c4-c363d35decc8","Handler":null,"Name":"","Endpoint":""} Aug 04 19:15:45.545592 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.545569 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 19:15:45.545730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.545612 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 19:15:45.603938 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.603915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:45.604071 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:45.604029 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:45.711019 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.710983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"3d5c8271df169eb95ce050f59619b7b7698f9ccdc793c4d5d396ae45ca6c376e"} Aug 04 19:15:45.711214 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.711075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"3f982dabcf0efc2bdc5b9bdeb2cbd3f3f545591c25f2ef70214a6ca668ea2b97"} Aug 04 19:15:45.711214 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.711129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"b8a689f2904f7557eaa43b9e74839f1cf8ffc28a3bfb0f2895700e6d1f28a95e"} Aug 04 19:15:45.711214 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.711179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"48e40d8f476028578148f1eb564f3ffd5069cd29fa9d9d8553d70db3e9215771"} Aug 04 19:15:45.711343 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.711224 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"618d15da8ed6cae8a2f5242ef8495dd59e31cb110bbf1a796d396291bebdab91"} Aug 04 19:15:45.711343 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.711273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"5cc6021f23ee1411e94b760afac816eb66b5316c5301749e2007b93c36ba7c61"} Aug 04 19:15:45.712634 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.712604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xcnvp" event={"ID":"209159ed-1897-4041-98cc-a925e9fe5306","Type":"ContainerStarted","Data":"0f4bada6586ae277eb125e2cf9643567259dd637582c00d3145840abb6bc1d75"} Aug 04 19:15:45.715377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.715297 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" event={"ID":"b340ed23-8473-48d2-be0a-53ef9b9ddb8c","Type":"ContainerStarted","Data":"ccd10f7280189177494a24b170b3744abb6f5e9661cfa930ea3d4c6ac0bc3108"} Aug 04 19:15:45.727845 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:45.727791 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xcnvp" podStartSLOduration=4.03102174 podStartE2EDuration="22.727774745s" totalImagesPullingTime="18.696753005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:45.727692133 +0000 UTC m=+22.668488878" watchObservedRunningTime="2026-08-04 19:15:45.727774745 +0000 UTC m=+22.668571485" Aug 04 19:15:46.604949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:46.604742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:46.605492 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:46.604773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:46.605492 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:46.605039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:46.605492 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:46.605157 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:47.604657 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:47.604622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:47.604950 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:47.604753 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:47.723668 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:47.723629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" event={"ID":"b340ed23-8473-48d2-be0a-53ef9b9ddb8c","Type":"ContainerStarted","Data":"06664f796beea430a48150c6fdd52d3db16b18933d972f2058b039c3f143def3"} Aug 04 19:15:47.746894 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:47.746839 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-789xq" podStartSLOduration=5.214210966 podStartE2EDuration="24.746824377s" totalImagesPullingTime="19.532613411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:47.745740595 +0000 UTC m=+24.686537342" watchObservedRunningTime="2026-08-04 19:15:47.746824377 +0000 UTC m=+24.687621122" Aug 04 19:15:48.603897 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:48.603855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:48.604134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:48.603869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:48.604134 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:48.603970 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:48.604134 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:48.604070 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:49.605234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:49.605203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:49.605726 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:49.605377 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:49.730130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:49.730096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerStarted","Data":"143a44390e6e6cab6c9c24943685b634e13e715af6ac029772124b2c077cd308"} Aug 04 19:15:49.733753 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:49.733715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"6b40c153afe96ee131018a7dee61d0de698e8a4f048f1401b07bd5a4c7473801"} Aug 04 19:15:50.604815 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:50.604777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:50.605021 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:50.604788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:50.605021 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:50.604886 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:50.605021 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:50.604980 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:50.737344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:50.737297 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e75a8e6-9463-4849-adaf-7b2cadc09529" containerID="143a44390e6e6cab6c9c24943685b634e13e715af6ac029772124b2c077cd308" exitCode=0 Aug 04 19:15:50.737706 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:50.737368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerDied","Data":"143a44390e6e6cab6c9c24943685b634e13e715af6ac029772124b2c077cd308"} Aug 04 19:15:51.604485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.604449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:51.604648 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:51.604574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:51.741947 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.741698 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e75a8e6-9463-4849-adaf-7b2cadc09529" containerID="eec3218170e8f6dbb5b11c88f20be82d1da45d0a4abdd6ef5217d59d806c5cea" exitCode=0 Aug 04 19:15:51.742689 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.741772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerDied","Data":"eec3218170e8f6dbb5b11c88f20be82d1da45d0a4abdd6ef5217d59d806c5cea"} Aug 04 19:15:51.746289 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.746262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" event={"ID":"66dbcaea-5165-425d-9573-caddd6f40f06","Type":"ContainerStarted","Data":"6bfdb9e8530eedbd3f5be9a9e0cc751624e7c78584f953feaff519e844ff354a"} Aug 04 19:15:51.746538 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.746517 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:51.746629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.746594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:51.746687 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.746656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:51.762690 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.762665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:51.762818 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.762764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:15:51.789860 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:51.789810 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" podStartSLOduration=10.036110815 podStartE2EDuration="28.789795468s" totalImagesPullingTime="18.753684653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:15:51.789112681 +0000 UTC m=+28.729909426" watchObservedRunningTime="2026-08-04 19:15:51.789795468 +0000 UTC m=+28.730592214" Aug 04 19:15:52.604190 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:52.604152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:52.604419 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:52.604158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:52.604419 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:52.604277 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:52.604419 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:52.604344 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:52.751529 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:52.751501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerStarted","Data":"0c6b09817fd8c212ff10193264450dd93c82f01f0f2373c466dc4be237314f6e"} Aug 04 19:15:53.544318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.544284 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qnd99"] Aug 04 19:15:53.544528 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.544491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:53.544651 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:53.544624 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:53.546055 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.545952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glb96"] Aug 04 19:15:53.546055 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.546054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:53.546253 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:53.546140 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:53.547801 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.547778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mm924"] Aug 04 19:15:53.547888 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.547872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:53.548045 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:53.548021 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:53.755774 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.755741 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e75a8e6-9463-4849-adaf-7b2cadc09529" containerID="0c6b09817fd8c212ff10193264450dd93c82f01f0f2373c466dc4be237314f6e" exitCode=0 Aug 04 19:15:53.756127 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:53.755815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerDied","Data":"0c6b09817fd8c212ff10193264450dd93c82f01f0f2373c466dc4be237314f6e"} Aug 04 19:15:55.604074 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:55.604036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:55.604074 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:55.604065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:55.604561 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:55.604037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:55.604561 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:55.604196 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnd99" podUID="2a8860b7-1f1b-4269-b469-cac0bc30c406" Aug 04 19:15:55.604561 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:55.604240 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glb96" podUID="4c10f094-a78b-4733-b541-57206eefdc6f" Aug 04 19:15:55.604561 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:55.604320 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm924" podUID="92e0a43b-f828-47fc-86af-81f2dbd29877" Aug 04 19:15:56.858597 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.858517 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-62.ec2.internal" event="NodeReady" Aug 04 19:15:56.859039 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.858655 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 19:15:56.893291 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.893261 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2"] Aug 04 19:15:56.975630 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.975590 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76"] Aug 04 19:15:56.975802 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.975727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:56.978086 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.978055 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 19:15:56.978086 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.978078 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:56.978272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.978089 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:56.978272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.978124 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5g7wr\"" Aug 04 19:15:56.989864 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.989706 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f95bbcb8-48t5s"] Aug 04 19:15:56.989973 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.989813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" Aug 04 19:15:56.993072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.993019 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:56.993244 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.993224 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ktww9\"" Aug 04 19:15:56.994968 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:56.994524 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.003795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.003773 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-cgnz7"] Aug 04 19:15:57.003902 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.003878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.006114 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.006096 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 19:15:57.006207 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.006055 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 19:15:57.006207 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.006198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 19:15:57.006457 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.006443 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5pc5\"" Aug 04 19:15:57.012504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.012486 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 19:15:57.024340 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.024300 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9"] Aug 04 19:15:57.024465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.024448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.026977 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.026954 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 19:15:57.027089 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.027030 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 19:15:57.027089 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.027045 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.027187 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.027157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h96vt\"" Aug 04 19:15:57.027187 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.027178 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.032659 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.032639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 19:15:57.036663 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.036289 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn"] Aug 04 19:15:57.036992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.036972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.039215 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.039185 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 19:15:57.039502 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.039487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.039565 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.039530 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sf9g9\"" Aug 04 19:15:57.039785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.039770 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 19:15:57.039847 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.039773 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.052814 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.052776 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2"] Aug 04 19:15:57.052814 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.052823 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76"] Aug 04 19:15:57.053023 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.052845 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p589j"] Aug 04 19:15:57.053023 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.052897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.055345 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.055309 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.055472 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.055310 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-85fsj\"" Aug 04 19:15:57.055530 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.055517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 19:15:57.055577 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.055542 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.055577 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.055568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 19:15:57.066507 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.066486 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc"] Aug 04 19:15:57.066640 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.066613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.069221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.069134 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 19:15:57.069221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.069157 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 19:15:57.069221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.069197 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svf4z\"" Aug 04 19:15:57.084749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.084716 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn"] Aug 04 19:15:57.084889 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.084760 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f95bbcb8-48t5s"] Aug 04 19:15:57.084889 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.084785 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bvf4l"] Aug 04 19:15:57.084889 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.084842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" Aug 04 19:15:57.087691 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.087666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-prfwf\"" Aug 04 19:15:57.087821 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.087729 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.087821 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.087753 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.109274 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.109190 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9"] Aug 04 19:15:57.109274 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.109229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc"] Aug 04 19:15:57.109607 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.109323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.109607 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.109348 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p589j"] Aug 04 19:15:57.109607 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.109457 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-cgnz7"] Aug 04 19:15:57.109607 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.109477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bvf4l"] Aug 04 19:15:57.112231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.112198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 19:15:57.112231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.112213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sds9s\"" Aug 04 19:15:57.112231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.112233 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 19:15:57.112576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.112560 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 19:15:57.135283 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47ff5264-2f50-43ed-ad20-820410bb85c7-config\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.135493 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/526fc512-89e6-4817-8d17-b561d2c1e7c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.135493 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t94f\" (UniqueName: \"kubernetes.io/projected/5d58d284-9cd3-40f7-bc60-3116bc0cc32f-kube-api-access-5t94f\") pod \"volume-data-source-validator-5b8776669-cpb76\" (UID: \"5d58d284-9cd3-40f7-bc60-3116bc0cc32f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" Aug 04 19:15:57.135493 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:57.135493 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77c67\" (UniqueName: \"kubernetes.io/projected/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-kube-api-access-77c67\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-trusted-ca\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-ca-trust-extracted\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fb8\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-kube-api-access-49fb8\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47ff5264-2f50-43ed-ad20-820410bb85c7-trusted-ca\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b279966b-352c-4362-9465-2bc0b9da85e5-config\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-certificates\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/526fc512-89e6-4817-8d17-b561d2c1e7c9-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.135702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjzk\" (UniqueName: \"kubernetes.io/projected/526fc512-89e6-4817-8d17-b561d2c1e7c9-kube-api-access-cwjzk\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47ff5264-2f50-43ed-ad20-820410bb85c7-serving-cert\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/47ff5264-2f50-43ed-ad20-820410bb85c7-kube-api-access-sz6l5\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-image-registry-private-configuration\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-installation-pull-secrets\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-bound-sa-token\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b279966b-352c-4362-9465-2bc0b9da85e5-serving-cert\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbcr\" (UniqueName: \"kubernetes.io/projected/b279966b-352c-4362-9465-2bc0b9da85e5-kube-api-access-jkbcr\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.136130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.135963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.236629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b279966b-352c-4362-9465-2bc0b9da85e5-config\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.236629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47ff5264-2f50-43ed-ad20-820410bb85c7-serving-cert\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.236854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/47ff5264-2f50-43ed-ad20-820410bb85c7-kube-api-access-sz6l5\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.236854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fb8\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-kube-api-access-49fb8\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.236854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-installation-pull-secrets\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.237060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d50fb28-2b17-4db2-8465-dc86997e17ec-config-volume\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.237060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b279966b-352c-4362-9465-2bc0b9da85e5-serving-cert\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.237060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.237060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.236980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t94f\" (UniqueName: \"kubernetes.io/projected/5d58d284-9cd3-40f7-bc60-3116bc0cc32f-kube-api-access-5t94f\") pod \"volume-data-source-validator-5b8776669-cpb76\" (UID: \"5d58d284-9cd3-40f7-bc60-3116bc0cc32f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" Aug 04 19:15:57.237060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47ff5264-2f50-43ed-ad20-820410bb85c7-config\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.237060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77c67\" (UniqueName: \"kubernetes.io/projected/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-kube-api-access-77c67\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-trusted-ca\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47ff5264-2f50-43ed-ad20-820410bb85c7-trusted-ca\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-certificates\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/526fc512-89e6-4817-8d17-b561d2c1e7c9-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjzk\" (UniqueName: \"kubernetes.io/projected/526fc512-89e6-4817-8d17-b561d2c1e7c9-kube-api-access-cwjzk\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7-cert\") pod \"ingress-canary-bvf4l\" (UID: \"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7\") " pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b279966b-352c-4362-9465-2bc0b9da85e5-config\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.237318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-ca-trust-extracted\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.237782 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.237373 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 19:15:57.237782 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.237454 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls podName:b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.73743434 +0000 UTC m=+34.678231079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-9cvt2" (UID: "b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc") : secret "samples-operator-tls" not found Aug 04 19:15:57.237782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-ca-trust-extracted\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.237938 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.237917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47ff5264-2f50-43ed-ad20-820410bb85c7-config\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.238245 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-certificates\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.238245 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47ff5264-2f50-43ed-ad20-820410bb85c7-trusted-ca\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.238245 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-image-registry-private-configuration\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.238420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-bound-sa-token\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.238420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d50fb28-2b17-4db2-8465-dc86997e17ec-tmp-dir\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.238420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5vrl\" (UniqueName: \"kubernetes.io/projected/3b30b26c-530f-4181-ae88-a6fa65b1b133-kube-api-access-v5vrl\") pod \"network-check-source-65bdbbff45-8zhmc\" (UID: \"3b30b26c-530f-4181-ae88-a6fa65b1b133\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" Aug 04 19:15:57.238420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbcr\" (UniqueName: \"kubernetes.io/projected/b279966b-352c-4362-9465-2bc0b9da85e5-kube-api-access-jkbcr\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.238586 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.238576 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 19:15:57.238630 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.238590 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f95bbcb8-48t5s: secret "image-registry-tls" not found Aug 04 19:15:57.238676 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.238639 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls podName:08cde8e9-373b-4d3c-9ec1-a970972ad1d7 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.738624813 +0000 UTC m=+34.679421552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls") pod "image-registry-68f95bbcb8-48t5s" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7") : secret "image-registry-tls" not found Aug 04 19:15:57.238676 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/526fc512-89e6-4817-8d17-b561d2c1e7c9-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.238822 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/526fc512-89e6-4817-8d17-b561d2c1e7c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.238923 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj46q\" (UniqueName: \"kubernetes.io/projected/6d50fb28-2b17-4db2-8465-dc86997e17ec-kube-api-access-qj46q\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.239007 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238984 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-trusted-ca\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.239059 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.238999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp9cc\" (UniqueName: \"kubernetes.io/projected/65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7-kube-api-access-gp9cc\") pod \"ingress-canary-bvf4l\" (UID: \"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7\") " pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.241632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.241602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/526fc512-89e6-4817-8d17-b561d2c1e7c9-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.241632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.241611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b279966b-352c-4362-9465-2bc0b9da85e5-serving-cert\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.241830 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.241719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47ff5264-2f50-43ed-ad20-820410bb85c7-serving-cert\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.241830 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.241794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-image-registry-private-configuration\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.241908 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.241835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-installation-pull-secrets\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.248520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.248473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fb8\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-kube-api-access-49fb8\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.250462 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.249263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6l5\" (UniqueName: \"kubernetes.io/projected/47ff5264-2f50-43ed-ad20-820410bb85c7-kube-api-access-sz6l5\") pod \"console-operator-5ccd6b445f-cgnz7\" (UID: \"47ff5264-2f50-43ed-ad20-820410bb85c7\") " pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.250462 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.249618 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjzk\" (UniqueName: \"kubernetes.io/projected/526fc512-89e6-4817-8d17-b561d2c1e7c9-kube-api-access-cwjzk\") pod \"kube-storage-version-migrator-operator-7878f4b88c-nlhhn\" (UID: \"526fc512-89e6-4817-8d17-b561d2c1e7c9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.250462 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.249959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t94f\" (UniqueName: \"kubernetes.io/projected/5d58d284-9cd3-40f7-bc60-3116bc0cc32f-kube-api-access-5t94f\") pod \"volume-data-source-validator-5b8776669-cpb76\" (UID: \"5d58d284-9cd3-40f7-bc60-3116bc0cc32f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" Aug 04 19:15:57.250462 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.250419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77c67\" (UniqueName: \"kubernetes.io/projected/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-kube-api-access-77c67\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:57.250713 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.250661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-bound-sa-token\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.251516 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.251488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbcr\" (UniqueName: \"kubernetes.io/projected/b279966b-352c-4362-9465-2bc0b9da85e5-kube-api-access-jkbcr\") pod \"service-ca-operator-784f97bcf6-fg8t9\" (UID: \"b279966b-352c-4362-9465-2bc0b9da85e5\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.301226 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.301192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" Aug 04 19:15:57.334774 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.334732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:15:57.340046 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:57.340240 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d50fb28-2b17-4db2-8465-dc86997e17ec-config-volume\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.340240 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7-cert\") pod \"ingress-canary-bvf4l\" (UID: \"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7\") " pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.340240 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.340181 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:57.340240 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.340252 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret podName:4c10f094-a78b-4733-b541-57206eefdc6f nodeName:}" failed. No retries permitted until 2026-08-04 19:16:29.340232974 +0000 UTC m=+66.281029719 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret") pod "global-pull-secret-syncer-glb96" (UID: "4c10f094-a78b-4733-b541-57206eefdc6f") : object "kube-system"/"original-pull-secret" not registered Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.340311 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d50fb28-2b17-4db2-8465-dc86997e17ec-tmp-dir\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5vrl\" (UniqueName: \"kubernetes.io/projected/3b30b26c-530f-4181-ae88-a6fa65b1b133-kube-api-access-v5vrl\") pod \"network-check-source-65bdbbff45-8zhmc\" (UID: \"3b30b26c-530f-4181-ae88-a6fa65b1b133\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj46q\" (UniqueName: \"kubernetes.io/projected/6d50fb28-2b17-4db2-8465-dc86997e17ec-kube-api-access-qj46q\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.340412 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls podName:6d50fb28-2b17-4db2-8465-dc86997e17ec nodeName:}" failed. No retries permitted until 2026-08-04 19:15:57.840400461 +0000 UTC m=+34.781197199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls") pod "dns-default-p589j" (UID: "6d50fb28-2b17-4db2-8465-dc86997e17ec") : secret "dns-default-metrics-tls" not found Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:57.340453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp9cc\" (UniqueName: \"kubernetes.io/projected/65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7-kube-api-access-gp9cc\") pod \"ingress-canary-bvf4l\" (UID: \"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7\") " pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.340852 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.340657 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:57.340852 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.340694 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs podName:2a8860b7-1f1b-4269-b469-cac0bc30c406 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:29.340682182 +0000 UTC m=+66.281478923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs") pod "network-metrics-daemon-qnd99" (UID: "2a8860b7-1f1b-4269-b469-cac0bc30c406") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 19:15:57.340852 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d50fb28-2b17-4db2-8465-dc86997e17ec-config-volume\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.340852 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.340838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d50fb28-2b17-4db2-8465-dc86997e17ec-tmp-dir\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.343207 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.343153 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7-cert\") pod \"ingress-canary-bvf4l\" (UID: \"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7\") " pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.343501 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.343482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dp2\" (UniqueName: \"kubernetes.io/projected/92e0a43b-f828-47fc-86af-81f2dbd29877-kube-api-access-24dp2\") pod \"network-check-target-mm924\" (UID: \"92e0a43b-f828-47fc-86af-81f2dbd29877\") " pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:57.346004 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.345979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" Aug 04 19:15:57.348883 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.348859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj46q\" (UniqueName: \"kubernetes.io/projected/6d50fb28-2b17-4db2-8465-dc86997e17ec-kube-api-access-qj46q\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.348971 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.348861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp9cc\" (UniqueName: \"kubernetes.io/projected/65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7-kube-api-access-gp9cc\") pod \"ingress-canary-bvf4l\" (UID: \"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7\") " pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.349305 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.349105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5vrl\" (UniqueName: \"kubernetes.io/projected/3b30b26c-530f-4181-ae88-a6fa65b1b133-kube-api-access-v5vrl\") pod \"network-check-source-65bdbbff45-8zhmc\" (UID: \"3b30b26c-530f-4181-ae88-a6fa65b1b133\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" Aug 04 19:15:57.363053 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.362984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" Aug 04 19:15:57.395015 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.394981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" Aug 04 19:15:57.420835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.420795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bvf4l" Aug 04 19:15:57.604128 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.604090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:15:57.604351 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.604091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:57.604351 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.604289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:15:57.606993 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.606970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 19:15:57.606993 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.606982 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 19:15:57.607189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.606994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bhjx\"" Aug 04 19:15:57.607189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.607048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pr7h8\"" Aug 04 19:15:57.622726 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.622701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:15:57.743775 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.743735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:57.743984 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.743793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:57.743984 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.743895 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 19:15:57.743984 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.743914 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f95bbcb8-48t5s: secret "image-registry-tls" not found Aug 04 19:15:57.743984 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.743915 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 19:15:57.743984 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.743973 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls podName:08cde8e9-373b-4d3c-9ec1-a970972ad1d7 nodeName:}" failed. No retries permitted until 2026-08-04 19:15:58.743953879 +0000 UTC m=+35.684750624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls") pod "image-registry-68f95bbcb8-48t5s" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7") : secret "image-registry-tls" not found Aug 04 19:15:57.744207 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.743992 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls podName:b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc nodeName:}" failed. No retries permitted until 2026-08-04 19:15:58.743982826 +0000 UTC m=+35.684779553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-9cvt2" (UID: "b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc") : secret "samples-operator-tls" not found Aug 04 19:15:57.845035 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:57.844993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:57.845230 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.845141 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 19:15:57.845230 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:57.845219 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls podName:6d50fb28-2b17-4db2-8465-dc86997e17ec nodeName:}" failed. No retries permitted until 2026-08-04 19:15:58.845198663 +0000 UTC m=+35.785995400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls") pod "dns-default-p589j" (UID: "6d50fb28-2b17-4db2-8465-dc86997e17ec") : secret "dns-default-metrics-tls" not found Aug 04 19:15:58.101514 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.101470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" limit=1024 Aug 04 19:15:58.194155 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.194112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" limit=1024 Aug 04 19:15:58.252314 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.252243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" limit=1024 Aug 04 19:15:58.754837 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.754798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:15:58.755034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.754848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:15:58.755034 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.755012 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 19:15:58.755151 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.755077 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls podName:b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc nodeName:}" failed. No retries permitted until 2026-08-04 19:16:00.75505849 +0000 UTC m=+37.695855232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-9cvt2" (UID: "b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc") : secret "samples-operator-tls" not found Aug 04 19:15:58.756068 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.755881 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 19:15:58.756068 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.755903 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f95bbcb8-48t5s: secret "image-registry-tls" not found Aug 04 19:15:58.756068 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.755955 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls podName:08cde8e9-373b-4d3c-9ec1-a970972ad1d7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:00.755938486 +0000 UTC m=+37.696735230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls") pod "image-registry-68f95bbcb8-48t5s" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7") : secret "image-registry-tls" not found Aug 04 19:15:58.858097 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.856771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:15:58.858097 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.856889 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 19:15:58.858097 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:15:58.856974 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls podName:6d50fb28-2b17-4db2-8465-dc86997e17ec nodeName:}" failed. No retries permitted until 2026-08-04 19:16:00.856942604 +0000 UTC m=+37.797739348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls") pod "dns-default-p589j" (UID: "6d50fb28-2b17-4db2-8465-dc86997e17ec") : secret "dns-default-metrics-tls" not found Aug 04 19:15:58.889107 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.888625 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526fc512_89e6_4817_8d17_b561d2c1e7c9.slice/crio-9dbf92cf2175e73ed0b183348eb9b4612190864972dd2974d113206455f2a716 WatchSource:0}: Error finding container 9dbf92cf2175e73ed0b183348eb9b4612190864972dd2974d113206455f2a716: Status 404 returned error can't find the container with id 9dbf92cf2175e73ed0b183348eb9b4612190864972dd2974d113206455f2a716 Aug 04 19:15:58.892759 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.891521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn"] Aug 04 19:15:58.892759 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.892286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-cgnz7"] Aug 04 19:15:58.894152 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.894105 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ff5264_2f50_43ed_ad20_820410bb85c7.slice/crio-d4b55a61629551e9e6b3ac9f7d4fd0e5fc5d5814f481c39424de7f8ec2df5fae WatchSource:0}: Error finding container d4b55a61629551e9e6b3ac9f7d4fd0e5fc5d5814f481c39424de7f8ec2df5fae: Status 404 returned error can't find the container with id d4b55a61629551e9e6b3ac9f7d4fd0e5fc5d5814f481c39424de7f8ec2df5fae Aug 04 19:15:58.897521 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.897479 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9"] Aug 04 19:15:58.900683 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.900656 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb279966b_352c_4362_9465_2bc0b9da85e5.slice/crio-d8448a2aad85c559b696fba9c70b3f60ca73571494945d1e0c1c957f299b9ec1 WatchSource:0}: Error finding container d8448a2aad85c559b696fba9c70b3f60ca73571494945d1e0c1c957f299b9ec1: Status 404 returned error can't find the container with id d8448a2aad85c559b696fba9c70b3f60ca73571494945d1e0c1c957f299b9ec1 Aug 04 19:15:58.903173 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.903149 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bvf4l"] Aug 04 19:15:58.906060 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.906036 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fb9b7e_8c2b_44f4_8a03_0ca043ee40a7.slice/crio-2eb091af8cb7610dc4af0a1dc8b84f3a0194bf00b63126ca2278f8af6293331d WatchSource:0}: Error finding container 2eb091af8cb7610dc4af0a1dc8b84f3a0194bf00b63126ca2278f8af6293331d: Status 404 returned error can't find the container with id 2eb091af8cb7610dc4af0a1dc8b84f3a0194bf00b63126ca2278f8af6293331d Aug 04 19:15:58.908940 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.908922 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76"] Aug 04 19:15:58.912057 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.912034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc"] Aug 04 19:15:58.912385 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.912368 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d58d284_9cd3_40f7_bc60_3116bc0cc32f.slice/crio-6374954a3ba0bdef94e4ea8b995b64fe0e8020af717f74b8be245f556dff8eb2 WatchSource:0}: Error finding container 6374954a3ba0bdef94e4ea8b995b64fe0e8020af717f74b8be245f556dff8eb2: Status 404 returned error can't find the container with id 6374954a3ba0bdef94e4ea8b995b64fe0e8020af717f74b8be245f556dff8eb2 Aug 04 19:15:58.915062 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:58.914967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mm924"] Aug 04 19:15:58.915196 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.915174 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b30b26c_530f_4181_ae88_a6fa65b1b133.slice/crio-1806694a11634dfe0a36acec934327e842036e137897882a8fe5c9f8f38dde56 WatchSource:0}: Error finding container 1806694a11634dfe0a36acec934327e842036e137897882a8fe5c9f8f38dde56: Status 404 returned error can't find the container with id 1806694a11634dfe0a36acec934327e842036e137897882a8fe5c9f8f38dde56 Aug 04 19:15:58.918475 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:15:58.918449 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e0a43b_f828_47fc_86af_81f2dbd29877.slice/crio-1e79aeb125475e664cc99cb11bcd0baa8f1bfc574d9076eb4e94803be44165bf WatchSource:0}: Error finding container 1e79aeb125475e664cc99cb11bcd0baa8f1bfc574d9076eb4e94803be44165bf: Status 404 returned error can't find the container with id 1e79aeb125475e664cc99cb11bcd0baa8f1bfc574d9076eb4e94803be44165bf Aug 04 19:15:59.780470 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.780083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bvf4l" event={"ID":"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7","Type":"ContainerStarted","Data":"2eb091af8cb7610dc4af0a1dc8b84f3a0194bf00b63126ca2278f8af6293331d"} Aug 04 19:15:59.782750 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.782607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" event={"ID":"526fc512-89e6-4817-8d17-b561d2c1e7c9","Type":"ContainerStarted","Data":"9dbf92cf2175e73ed0b183348eb9b4612190864972dd2974d113206455f2a716"} Aug 04 19:15:59.784080 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.784014 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" event={"ID":"b279966b-352c-4362-9465-2bc0b9da85e5","Type":"ContainerStarted","Data":"d8448a2aad85c559b696fba9c70b3f60ca73571494945d1e0c1c957f299b9ec1"} Aug 04 19:15:59.790552 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.789564 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e75a8e6-9463-4849-adaf-7b2cadc09529" containerID="f132683db641755eb090dfa29427f5d8c32a27eb1ed49a3e89ca396ca8c0752d" exitCode=0 Aug 04 19:15:59.790552 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.789642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerDied","Data":"f132683db641755eb090dfa29427f5d8c32a27eb1ed49a3e89ca396ca8c0752d"} Aug 04 19:15:59.792436 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.792374 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" event={"ID":"47ff5264-2f50-43ed-ad20-820410bb85c7","Type":"ContainerStarted","Data":"d4b55a61629551e9e6b3ac9f7d4fd0e5fc5d5814f481c39424de7f8ec2df5fae"} Aug 04 19:15:59.794585 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.794305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mm924" event={"ID":"92e0a43b-f828-47fc-86af-81f2dbd29877","Type":"ContainerStarted","Data":"1e79aeb125475e664cc99cb11bcd0baa8f1bfc574d9076eb4e94803be44165bf"} Aug 04 19:15:59.796165 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.796075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" event={"ID":"3b30b26c-530f-4181-ae88-a6fa65b1b133","Type":"ContainerStarted","Data":"1806694a11634dfe0a36acec934327e842036e137897882a8fe5c9f8f38dde56"} Aug 04 19:15:59.797753 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:15:59.797606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" event={"ID":"5d58d284-9cd3-40f7-bc60-3116bc0cc32f","Type":"ContainerStarted","Data":"6374954a3ba0bdef94e4ea8b995b64fe0e8020af717f74b8be245f556dff8eb2"} Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:00.775249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:00.775305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.775483 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.775537 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls podName:b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc nodeName:}" failed. No retries permitted until 2026-08-04 19:16:04.775524041 +0000 UTC m=+41.716320766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-9cvt2" (UID: "b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc") : secret "samples-operator-tls" not found Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.775585 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.775593 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f95bbcb8-48t5s: secret "image-registry-tls" not found Aug 04 19:16:00.775900 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.775617 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls podName:08cde8e9-373b-4d3c-9ec1-a970972ad1d7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:04.775608968 +0000 UTC m=+41.716405692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls") pod "image-registry-68f95bbcb8-48t5s" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7") : secret "image-registry-tls" not found Aug 04 19:16:00.814972 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:00.814502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerStarted","Data":"2a04a37c49a97aeb7942e66dc465894da73301aba4826602911a4bcf9b4ddfcb"} Aug 04 19:16:00.876591 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:00.876366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:16:00.877301 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.876835 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 19:16:00.877301 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:00.876917 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls podName:6d50fb28-2b17-4db2-8465-dc86997e17ec nodeName:}" failed. No retries permitted until 2026-08-04 19:16:04.876896451 +0000 UTC m=+41.817693199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls") pod "dns-default-p589j" (UID: "6d50fb28-2b17-4db2-8465-dc86997e17ec") : secret "dns-default-metrics-tls" not found Aug 04 19:16:01.139792 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.139748 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf"] Aug 04 19:16:01.162052 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.162016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf"] Aug 04 19:16:01.162214 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.162150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.165132 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.165098 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 04 19:16:01.165476 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.165450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 04 19:16:01.165723 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.165706 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2f659\"" Aug 04 19:16:01.266708 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.266593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:01.281218 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.281176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.281452 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.281260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2709f187-9682-4546-ba3f-b8671e3f2ab7-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.369524 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.369481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:01.382148 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.382107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.382340 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.382195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2709f187-9682-4546-ba3f-b8671e3f2ab7-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.383544 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.383081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2709f187-9682-4546-ba3f-b8671e3f2ab7-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.383544 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:01.383209 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 19:16:01.383544 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:01.383270 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert podName:2709f187-9682-4546-ba3f-b8671e3f2ab7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:01.883251991 +0000 UTC m=+38.824048718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-x7ktf" (UID: "2709f187-9682-4546-ba3f-b8671e3f2ab7") : secret "networking-console-plugin-cert" not found Aug 04 19:16:01.469403 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.469275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" limit=1024 Aug 04 19:16:01.826595 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.826556 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e75a8e6-9463-4849-adaf-7b2cadc09529" containerID="2a04a37c49a97aeb7942e66dc465894da73301aba4826602911a4bcf9b4ddfcb" exitCode=0 Aug 04 19:16:01.827179 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.826629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerDied","Data":"2a04a37c49a97aeb7942e66dc465894da73301aba4826602911a4bcf9b4ddfcb"} Aug 04 19:16:01.888920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:01.888302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:01.888920 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:01.888497 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 19:16:01.888920 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:01.888563 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert podName:2709f187-9682-4546-ba3f-b8671e3f2ab7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:02.888543112 +0000 UTC m=+39.829339839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-x7ktf" (UID: "2709f187-9682-4546-ba3f-b8671e3f2ab7") : secret "networking-console-plugin-cert" not found Aug 04 19:16:02.898370 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:02.898318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:02.898865 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:02.898483 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 19:16:02.898865 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:02.898577 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert podName:2709f187-9682-4546-ba3f-b8671e3f2ab7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:04.898552778 +0000 UTC m=+41.839349503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-x7ktf" (UID: "2709f187-9682-4546-ba3f-b8671e3f2ab7") : secret "networking-console-plugin-cert" not found Aug 04 19:16:04.813780 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:04.813726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:04.814217 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:04.813790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:16:04.814217 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.813905 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 19:16:04.814217 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.813985 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls podName:b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc nodeName:}" failed. No retries permitted until 2026-08-04 19:16:12.813963939 +0000 UTC m=+49.754760683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-9cvt2" (UID: "b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc") : secret "samples-operator-tls" not found Aug 04 19:16:04.814217 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.813911 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 19:16:04.814217 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.814021 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f95bbcb8-48t5s: secret "image-registry-tls" not found Aug 04 19:16:04.814217 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.814079 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls podName:08cde8e9-373b-4d3c-9ec1-a970972ad1d7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:12.814061514 +0000 UTC m=+49.754858257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls") pod "image-registry-68f95bbcb8-48t5s" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7") : secret "image-registry-tls" not found Aug 04 19:16:04.914969 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:04.914931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:16:04.915175 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:04.914989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:04.915175 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.915117 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 19:16:04.915298 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.915200 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls podName:6d50fb28-2b17-4db2-8465-dc86997e17ec nodeName:}" failed. No retries permitted until 2026-08-04 19:16:12.915179001 +0000 UTC m=+49.855975744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls") pod "dns-default-p589j" (UID: "6d50fb28-2b17-4db2-8465-dc86997e17ec") : secret "dns-default-metrics-tls" not found Aug 04 19:16:04.915298 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.915126 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 19:16:04.915437 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:04.915310 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert podName:2709f187-9682-4546-ba3f-b8671e3f2ab7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:08.915289605 +0000 UTC m=+45.856086356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-x7ktf" (UID: "2709f187-9682-4546-ba3f-b8671e3f2ab7") : secret "networking-console-plugin-cert" not found Aug 04 19:16:06.842657 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.842385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" event={"ID":"5d58d284-9cd3-40f7-bc60-3116bc0cc32f","Type":"ContainerStarted","Data":"4070a028a09b6693f916863af4452cace802eb240c071f7a9ede81243077a751"} Aug 04 19:16:06.844034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.844012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bvf4l" event={"ID":"65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7","Type":"ContainerStarted","Data":"0886d98d20783479bdb2d25a9aa5089f0e2a6287044f167c6e5380043f957990"} Aug 04 19:16:06.845672 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.845646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" event={"ID":"526fc512-89e6-4817-8d17-b561d2c1e7c9","Type":"ContainerStarted","Data":"5631312a9754e7d0b0aced41d78e6704209283486257a31c643885ae3e9e4edc"} Aug 04 19:16:06.847252 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.847225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" event={"ID":"b279966b-352c-4362-9465-2bc0b9da85e5","Type":"ContainerStarted","Data":"6a36ea644e800d6b160fc61412cf45a95233915c6511aab77818ebafd47d8971"} Aug 04 19:16:06.851286 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.851068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" event={"ID":"1e75a8e6-9463-4849-adaf-7b2cadc09529","Type":"ContainerStarted","Data":"049f90b89373a700e85211253dd32ca66f0b96c2590ce342e7ef7eb7e12b7277"} Aug 04 19:16:06.852791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.852774 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/0.log" Aug 04 19:16:06.852889 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.852805 2579 generic.go:356] "Generic (PLEG): container finished" podID="47ff5264-2f50-43ed-ad20-820410bb85c7" containerID="7ee71bc7ae511803e9b79d86f7770f89659b3f9924b22a8ba3a7cbfc92161b3e" exitCode=255 Aug 04 19:16:06.852889 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.852831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" event={"ID":"47ff5264-2f50-43ed-ad20-820410bb85c7","Type":"ContainerDied","Data":"7ee71bc7ae511803e9b79d86f7770f89659b3f9924b22a8ba3a7cbfc92161b3e"} Aug 04 19:16:06.853076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.853053 2579 scope.go:117] "RemoveContainer" containerID="7ee71bc7ae511803e9b79d86f7770f89659b3f9924b22a8ba3a7cbfc92161b3e" Aug 04 19:16:06.854440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.854418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mm924" event={"ID":"92e0a43b-f828-47fc-86af-81f2dbd29877","Type":"ContainerStarted","Data":"d8e0c845b0eb771cee0496bb7779d94e96aaa6867a116f05b2be87c490bdf0f3"} Aug 04 19:16:06.854532 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.854479 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:16:06.856051 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.856009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" event={"ID":"3b30b26c-530f-4181-ae88-a6fa65b1b133","Type":"ContainerStarted","Data":"76eab562cd92b22ddc1fbf0b82ae93856462089ed4803c0ae2ce3dc725932296"} Aug 04 19:16:06.858924 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.858885 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-cpb76" podStartSLOduration=31.853613828 podStartE2EDuration="38.858871326s" totalImagesPullingTime="7.005257498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.858079022 +0000 UTC m=+43.798875768" watchObservedRunningTime="2026-08-04 19:16:06.858871326 +0000 UTC m=+43.799668064" Aug 04 19:16:06.878223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.878156 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" podStartSLOduration=28.624839688 podStartE2EDuration="35.878136055s" totalImagesPullingTime="7.253296367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.877241184 +0000 UTC m=+43.818037930" watchObservedRunningTime="2026-08-04 19:16:06.878136055 +0000 UTC m=+43.818932802" Aug 04 19:16:06.895093 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.895034 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-8zhmc" podStartSLOduration=24.551050987 podStartE2EDuration="31.895010054s" totalImagesPullingTime="7.343959067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.893441694 +0000 UTC m=+43.834238440" watchObservedRunningTime="2026-08-04 19:16:06.895010054 +0000 UTC m=+43.835806794" Aug 04 19:16:06.910403 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.910306 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bvf4l" podStartSLOduration=3.588784587 podStartE2EDuration="10.910285041s" totalImagesPullingTime="7.321500454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.909537014 +0000 UTC m=+43.850333764" watchObservedRunningTime="2026-08-04 19:16:06.910285041 +0000 UTC m=+43.851081785" Aug 04 19:16:06.949855 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.949800 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2sh5f" podStartSLOduration=19.294741353 podStartE2EDuration="43.949782494s" totalImagesPullingTime="24.655041141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.933687249 +0000 UTC m=+43.874483996" watchObservedRunningTime="2026-08-04 19:16:06.949782494 +0000 UTC m=+43.890579241" Aug 04 19:16:06.950134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.950111 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" podStartSLOduration=28.685730429 podStartE2EDuration="35.950105154s" totalImagesPullingTime="7.264374725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.948986186 +0000 UTC m=+43.889782932" watchObservedRunningTime="2026-08-04 19:16:06.950105154 +0000 UTC m=+43.890901899" Aug 04 19:16:06.986005 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:06.985937 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mm924" podStartSLOduration=36.633762283 podStartE2EDuration="43.985915307s" totalImagesPullingTime="7.352153024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:06.985683818 +0000 UTC m=+43.926480576" watchObservedRunningTime="2026-08-04 19:16:06.985915307 +0000 UTC m=+43.926712055" Aug 04 19:16:07.334934 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.334891 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:16:07.335134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.335000 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:16:07.861409 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.861383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:16:07.861875 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.861819 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/0.log" Aug 04 19:16:07.861875 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.861853 2579 generic.go:356] "Generic (PLEG): container finished" podID="47ff5264-2f50-43ed-ad20-820410bb85c7" containerID="28bd14a866086382a18900de00e8a8891dc2fa417419bf73cfd6bea9adcee34b" exitCode=255 Aug 04 19:16:07.861984 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.861891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" event={"ID":"47ff5264-2f50-43ed-ad20-820410bb85c7","Type":"ContainerDied","Data":"28bd14a866086382a18900de00e8a8891dc2fa417419bf73cfd6bea9adcee34b"} Aug 04 19:16:07.861984 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.861950 2579 scope.go:117] "RemoveContainer" containerID="7ee71bc7ae511803e9b79d86f7770f89659b3f9924b22a8ba3a7cbfc92161b3e" Aug 04 19:16:07.862650 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:07.862633 2579 scope.go:117] "RemoveContainer" containerID="28bd14a866086382a18900de00e8a8891dc2fa417419bf73cfd6bea9adcee34b" Aug 04 19:16:07.862977 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:07.862946 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-cgnz7_openshift-console-operator(47ff5264-2f50-43ed-ad20-820410bb85c7)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" podUID="47ff5264-2f50-43ed-ad20-820410bb85c7" Aug 04 19:16:08.867061 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:08.867035 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:16:08.867426 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:08.867375 2579 scope.go:117] "RemoveContainer" containerID="28bd14a866086382a18900de00e8a8891dc2fa417419bf73cfd6bea9adcee34b" Aug 04 19:16:08.867550 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:08.867531 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-cgnz7_openshift-console-operator(47ff5264-2f50-43ed-ad20-820410bb85c7)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" podUID="47ff5264-2f50-43ed-ad20-820410bb85c7" Aug 04 19:16:08.953933 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:08.953895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:08.954257 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:08.954230 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 19:16:08.954405 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:08.954391 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert podName:2709f187-9682-4546-ba3f-b8671e3f2ab7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:16.954302089 +0000 UTC m=+53.895098826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-x7ktf" (UID: "2709f187-9682-4546-ba3f-b8671e3f2ab7") : secret "networking-console-plugin-cert" not found Aug 04 19:16:09.637515 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.637481 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh"] Aug 04 19:16:09.665584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.665544 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh"] Aug 04 19:16:09.665729 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.665683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" Aug 04 19:16:09.668356 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.668308 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 04 19:16:09.669124 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.669099 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fc47n\"" Aug 04 19:16:09.669243 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.669098 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 04 19:16:09.760981 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.760943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vmt\" (UniqueName: \"kubernetes.io/projected/64f740d6-37be-4d0d-87f9-ff984fe09397-kube-api-access-65vmt\") pod \"migrator-68bc745d87-mfqxh\" (UID: \"64f740d6-37be-4d0d-87f9-ff984fe09397\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" Aug 04 19:16:09.861386 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.861358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vmt\" (UniqueName: \"kubernetes.io/projected/64f740d6-37be-4d0d-87f9-ff984fe09397-kube-api-access-65vmt\") pod \"migrator-68bc745d87-mfqxh\" (UID: \"64f740d6-37be-4d0d-87f9-ff984fe09397\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" Aug 04 19:16:09.872146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.872017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vmt\" (UniqueName: \"kubernetes.io/projected/64f740d6-37be-4d0d-87f9-ff984fe09397-kube-api-access-65vmt\") pod \"migrator-68bc745d87-mfqxh\" (UID: \"64f740d6-37be-4d0d-87f9-ff984fe09397\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" Aug 04 19:16:09.975802 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.975727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" Aug 04 19:16:09.983453 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:09.983423 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-crfdw"] Aug 04 19:16:10.005075 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.005040 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-crfdw"] Aug 04 19:16:10.005213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.005137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.007669 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.007644 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 04 19:16:10.007787 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.007680 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5rg4k\"" Aug 04 19:16:10.007847 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.007644 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 04 19:16:10.007847 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.007786 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 04 19:16:10.008082 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.008053 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 04 19:16:10.064272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.064199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42f88e65-7ee0-4758-afc1-9f435457d548-signing-key\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.064480 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.064299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42f88e65-7ee0-4758-afc1-9f435457d548-signing-cabundle\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.064480 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.064367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfdm\" (UniqueName: \"kubernetes.io/projected/42f88e65-7ee0-4758-afc1-9f435457d548-kube-api-access-bxfdm\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.127625 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.127590 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh"] Aug 04 19:16:10.131229 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:10.131197 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f740d6_37be_4d0d_87f9_ff984fe09397.slice/crio-a2b2895da2fc744e89ed8971a83aca884fce87017a2793aa95b582dd7e581024 WatchSource:0}: Error finding container a2b2895da2fc744e89ed8971a83aca884fce87017a2793aa95b582dd7e581024: Status 404 returned error can't find the container with id a2b2895da2fc744e89ed8971a83aca884fce87017a2793aa95b582dd7e581024 Aug 04 19:16:10.165019 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.164982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42f88e65-7ee0-4758-afc1-9f435457d548-signing-cabundle\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.165195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.165044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfdm\" (UniqueName: \"kubernetes.io/projected/42f88e65-7ee0-4758-afc1-9f435457d548-kube-api-access-bxfdm\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.165195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.165077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42f88e65-7ee0-4758-afc1-9f435457d548-signing-key\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.165769 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.165745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42f88e65-7ee0-4758-afc1-9f435457d548-signing-cabundle\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.168129 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.168103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42f88e65-7ee0-4758-afc1-9f435457d548-signing-key\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.176951 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.176924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfdm\" (UniqueName: \"kubernetes.io/projected/42f88e65-7ee0-4758-afc1-9f435457d548-kube-api-access-bxfdm\") pod \"service-ca-98c88fbb5-crfdw\" (UID: \"42f88e65-7ee0-4758-afc1-9f435457d548\") " pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.323047 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.322998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" Aug 04 19:16:10.476127 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.476092 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-crfdw"] Aug 04 19:16:10.479256 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:10.479232 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f88e65_7ee0_4758_afc1_9f435457d548.slice/crio-286d422efb369798a8518b8c18da4d4932021bbb0e532205f5acd745c1b50ec7 WatchSource:0}: Error finding container 286d422efb369798a8518b8c18da4d4932021bbb0e532205f5acd745c1b50ec7: Status 404 returned error can't find the container with id 286d422efb369798a8518b8c18da4d4932021bbb0e532205f5acd745c1b50ec7 Aug 04 19:16:10.875584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.875483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" event={"ID":"42f88e65-7ee0-4758-afc1-9f435457d548","Type":"ContainerStarted","Data":"7e2963138e0a6ee45a2fbe2a781d3712dd1e6e8483c1acce53296d99a7d8bd4f"} Aug 04 19:16:10.875584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.875550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" event={"ID":"42f88e65-7ee0-4758-afc1-9f435457d548","Type":"ContainerStarted","Data":"286d422efb369798a8518b8c18da4d4932021bbb0e532205f5acd745c1b50ec7"} Aug 04 19:16:10.876786 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.876761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" event={"ID":"64f740d6-37be-4d0d-87f9-ff984fe09397","Type":"ContainerStarted","Data":"a2b2895da2fc744e89ed8971a83aca884fce87017a2793aa95b582dd7e581024"} Aug 04 19:16:10.893888 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:10.893825 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-crfdw" podStartSLOduration=1.893808578 podStartE2EDuration="1.893808578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:10.892987719 +0000 UTC m=+47.833784464" watchObservedRunningTime="2026-08-04 19:16:10.893808578 +0000 UTC m=+47.834605324" Aug 04 19:16:12.885631 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:12.885587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" event={"ID":"64f740d6-37be-4d0d-87f9-ff984fe09397","Type":"ContainerStarted","Data":"c042dce8d751301c9e797f19c038d5f228ed90aee701d4786e85778388525ed4"} Aug 04 19:16:12.886036 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:12.885652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" event={"ID":"64f740d6-37be-4d0d-87f9-ff984fe09397","Type":"ContainerStarted","Data":"2ebe6648dfabf28dd2fc78df61b13c08cb36a980bae3baf4fabcf702536515ca"} Aug 04 19:16:12.889132 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:12.889097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:12.889261 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:12.889138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:16:12.889368 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.889286 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 19:16:12.889368 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.889299 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f95bbcb8-48t5s: secret "image-registry-tls" not found Aug 04 19:16:12.889368 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.889312 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 19:16:12.889368 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.889362 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls podName:08cde8e9-373b-4d3c-9ec1-a970972ad1d7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:28.889344864 +0000 UTC m=+65.830141605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls") pod "image-registry-68f95bbcb8-48t5s" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7") : secret "image-registry-tls" not found Aug 04 19:16:12.889566 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.889377 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls podName:b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc nodeName:}" failed. No retries permitted until 2026-08-04 19:16:28.889371036 +0000 UTC m=+65.830167760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-9cvt2" (UID: "b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc") : secret "samples-operator-tls" not found Aug 04 19:16:12.902858 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:12.902795 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mfqxh" podStartSLOduration=1.9632514319999999 podStartE2EDuration="3.90276924s" totalImagesPullingTime="1.939517808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:12.900560113 +0000 UTC m=+49.841356924" watchObservedRunningTime="2026-08-04 19:16:12.90276924 +0000 UTC m=+49.843565986" Aug 04 19:16:12.990412 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:12.990379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:16:12.990579 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.990531 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 19:16:12.990634 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:12.990603 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls podName:6d50fb28-2b17-4db2-8465-dc86997e17ec nodeName:}" failed. No retries permitted until 2026-08-04 19:16:28.990585793 +0000 UTC m=+65.931382530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls") pod "dns-default-p589j" (UID: "6d50fb28-2b17-4db2-8465-dc86997e17ec") : secret "dns-default-metrics-tls" not found Aug 04 19:16:17.028914 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.028869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:17.029485 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:17.029027 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 19:16:17.029485 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:17.029099 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert podName:2709f187-9682-4546-ba3f-b8671e3f2ab7 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:33.029079495 +0000 UTC m=+69.969876251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-x7ktf" (UID: "2709f187-9682-4546-ba3f-b8671e3f2ab7") : secret "networking-console-plugin-cert" not found Aug 04 19:16:17.335203 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.335163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:16:17.335416 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.335270 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:16:17.335701 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.335683 2579 scope.go:117] "RemoveContainer" containerID="28bd14a866086382a18900de00e8a8891dc2fa417419bf73cfd6bea9adcee34b" Aug 04 19:16:17.903450 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.903423 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:16:17.903618 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.903479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" event={"ID":"47ff5264-2f50-43ed-ad20-820410bb85c7","Type":"ContainerStarted","Data":"4faad9721dee8c83a7b61a1c3ff23621a95efaf83822f1a482f1b7d0499a5b50"} Aug 04 19:16:17.903775 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.903754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:16:17.908771 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.908749 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" Aug 04 19:16:17.927293 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:17.927244 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-cgnz7" podStartSLOduration=41.667891233 podStartE2EDuration="48.927229816s" totalImagesPullingTime="7.259338583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:17.926871099 +0000 UTC m=+54.867667876" watchObservedRunningTime="2026-08-04 19:16:17.927229816 +0000 UTC m=+54.868026562" Aug 04 19:16:23.767463 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:23.767434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kml" Aug 04 19:16:28.933860 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:28.933811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:28.933860 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:28.933864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:16:28.936438 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:28.936405 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"image-registry-68f95bbcb8-48t5s\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:28.936567 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:28.936469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-9cvt2\" (UID: \"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:16:29.035107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.035065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:16:29.037459 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.037432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d50fb28-2b17-4db2-8465-dc86997e17ec-metrics-tls\") pod \"dns-default-p589j\" (UID: \"6d50fb28-2b17-4db2-8465-dc86997e17ec\") " pod="openshift-dns/dns-default-p589j" Aug 04 19:16:29.087964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.087932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5g7wr\"" Aug 04 19:16:29.095754 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.095728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" Aug 04 19:16:29.115693 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.115658 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5pc5\"" Aug 04 19:16:29.123491 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.123467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:29.182779 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.182703 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svf4z\"" Aug 04 19:16:29.191520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.191431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p589j" Aug 04 19:16:29.270425 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.269924 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2"] Aug 04 19:16:29.291675 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.291642 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f95bbcb8-48t5s"] Aug 04 19:16:29.295499 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:29.295474 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cde8e9_373b_4d3c_9ec1_a970972ad1d7.slice/crio-059e92b4064fe8174495e24eeff70fd9ef7b3080e7420f7794259b2b11b6e19a WatchSource:0}: Error finding container 059e92b4064fe8174495e24eeff70fd9ef7b3080e7420f7794259b2b11b6e19a: Status 404 returned error can't find the container with id 059e92b4064fe8174495e24eeff70fd9ef7b3080e7420f7794259b2b11b6e19a Aug 04 19:16:29.350656 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.350630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p589j"] Aug 04 19:16:29.356400 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:29.356369 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d50fb28_2b17_4db2_8465_dc86997e17ec.slice/crio-7a2d0cb69a2455e89175c6f9719c8f38dad09fc3fab45a971724a193355a09aa WatchSource:0}: Error finding container 7a2d0cb69a2455e89175c6f9719c8f38dad09fc3fab45a971724a193355a09aa: Status 404 returned error can't find the container with id 7a2d0cb69a2455e89175c6f9719c8f38dad09fc3fab45a971724a193355a09aa Aug 04 19:16:29.438124 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.438092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:16:29.438278 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.438144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:16:29.440509 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.440489 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 19:16:29.440557 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.440490 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 19:16:29.450646 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.450578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8860b7-1f1b-4269-b469-cac0bc30c406-metrics-certs\") pod \"network-metrics-daemon-qnd99\" (UID: \"2a8860b7-1f1b-4269-b469-cac0bc30c406\") " pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:16:29.450646 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.450617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4c10f094-a78b-4733-b541-57206eefdc6f-original-pull-secret\") pod \"global-pull-secret-syncer-glb96\" (UID: \"4c10f094-a78b-4733-b541-57206eefdc6f\") " pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:16:29.719402 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.719302 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pr7h8\"" Aug 04 19:16:29.727173 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.727153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnd99" Aug 04 19:16:29.729377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.729351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glb96" Aug 04 19:16:29.893697 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.893605 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qnd99"] Aug 04 19:16:29.897291 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:29.897263 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8860b7_1f1b_4269_b469_cac0bc30c406.slice/crio-fb7033f42b9b7f12a370addbb69e3757d0f2937357b6caef36c52ed07b85f23e WatchSource:0}: Error finding container fb7033f42b9b7f12a370addbb69e3757d0f2937357b6caef36c52ed07b85f23e: Status 404 returned error can't find the container with id fb7033f42b9b7f12a370addbb69e3757d0f2937357b6caef36c52ed07b85f23e Aug 04 19:16:29.914659 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.914628 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glb96"] Aug 04 19:16:29.917701 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:29.917673 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c10f094_a78b_4733_b541_57206eefdc6f.slice/crio-9fc180d0b01e8791b5ba73c5fe78509a126ea1a2e6e55cb9960896e26455bd1f WatchSource:0}: Error finding container 9fc180d0b01e8791b5ba73c5fe78509a126ea1a2e6e55cb9960896e26455bd1f: Status 404 returned error can't find the container with id 9fc180d0b01e8791b5ba73c5fe78509a126ea1a2e6e55cb9960896e26455bd1f Aug 04 19:16:29.947241 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.947200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnd99" event={"ID":"2a8860b7-1f1b-4269-b469-cac0bc30c406","Type":"ContainerStarted","Data":"fb7033f42b9b7f12a370addbb69e3757d0f2937357b6caef36c52ed07b85f23e"} Aug 04 19:16:29.948958 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.948923 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" event={"ID":"08cde8e9-373b-4d3c-9ec1-a970972ad1d7","Type":"ContainerStarted","Data":"f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109"} Aug 04 19:16:29.949157 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.949117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" event={"ID":"08cde8e9-373b-4d3c-9ec1-a970972ad1d7","Type":"ContainerStarted","Data":"059e92b4064fe8174495e24eeff70fd9ef7b3080e7420f7794259b2b11b6e19a"} Aug 04 19:16:29.950658 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.950631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" event={"ID":"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc","Type":"ContainerStarted","Data":"b0c00e69f888917068b26def54c7ce299122f6610b6e8166d9e1490bb64ba751"} Aug 04 19:16:29.950789 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.950761 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6dd5c49894-ktdqh"] Aug 04 19:16:29.954628 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.954607 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-rd8q2"] Aug 04 19:16:29.954805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.954787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:29.956692 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.956674 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9"] Aug 04 19:16:29.956831 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.956813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:29.957319 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.957294 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 19:16:29.957517 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.957347 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 19:16:29.957652 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.957636 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 19:16:29.957813 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.957796 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 19:16:29.958386 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.958367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7dnnv\"" Aug 04 19:16:29.958630 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.958616 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz"] Aug 04 19:16:29.958691 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.958628 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 19:16:29.958691 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.958675 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 19:16:29.958793 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.958741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:29.960275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.960257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glb96" event={"ID":"4c10f094-a78b-4733-b541-57206eefdc6f","Type":"ContainerStarted","Data":"9fc180d0b01e8791b5ba73c5fe78509a126ea1a2e6e55cb9960896e26455bd1f"} Aug 04 19:16:29.960353 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.960282 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5"] Aug 04 19:16:29.960394 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.960368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:29.961094 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.961077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 19:16:29.961184 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.961157 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 19:16:29.961714 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.961694 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 19:16:29.962191 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p589j" event={"ID":"6d50fb28-2b17-4db2-8465-dc86997e17ec","Type":"ContainerStarted","Data":"7a2d0cb69a2455e89175c6f9719c8f38dad09fc3fab45a971724a193355a09aa"} Aug 04 19:16:29.962256 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:29.962401 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962380 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kkwh8\"" Aug 04 19:16:29.962401 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962399 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7zmjt\"" Aug 04 19:16:29.962530 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962474 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 19:16:29.962817 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 19:16:29.962931 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962915 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 19:16:29.962967 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.962949 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 19:16:29.963065 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.963054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 19:16:29.963132 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.963108 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 04 19:16:29.963222 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.963147 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 19:16:29.963554 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.963382 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 19:16:29.963644 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.963623 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 19:16:29.967051 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.967029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 19:16:29.967146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.967029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 19:16:29.967351 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.967322 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 19:16:29.967438 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.967387 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 19:16:29.969786 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.969731 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz"] Aug 04 19:16:29.972929 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.972910 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 19:16:29.979239 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.979169 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9"] Aug 04 19:16:29.982777 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.979709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-rd8q2"] Aug 04 19:16:29.982777 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.982538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dd5c49894-ktdqh"] Aug 04 19:16:29.983244 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.983117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" podStartSLOduration=66.983099483 podStartE2EDuration="1m6.983099483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:29.980099442 +0000 UTC m=+66.920896189" watchObservedRunningTime="2026-08-04 19:16:29.983099483 +0000 UTC m=+66.923896230" Aug 04 19:16:29.995718 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.995691 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z"] Aug 04 19:16:29.998181 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.998160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:29.998382 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:29.998359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5"] Aug 04 19:16:30.001603 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.001580 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 04 19:16:30.001730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.001702 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-727xj\"" Aug 04 19:16:30.015578 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.015548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z"] Aug 04 19:16:30.040732 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.040702 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-vtn4d"] Aug 04 19:16:30.042692 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.042675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:30.044402 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s97w\" (UniqueName: \"kubernetes.io/projected/6f6e778e-c311-4bab-b5f4-b4614e3b7795-kube-api-access-5s97w\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.044511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcm4\" (UniqueName: \"kubernetes.io/projected/0e808b6b-cba6-4664-9833-345b39b4f97f-kube-api-access-fqcm4\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.044581 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.044623 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-default-certificate\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.044664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0092bf4b-84ef-4574-9726-4ed506b345c6-snapshots\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.044704 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhql8\" (UniqueName: \"kubernetes.io/projected/0092bf4b-84ef-4574-9726-4ed506b345c6-kube-api-access-zhql8\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.044742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-kube-api-access-nkz78\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.044779 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-ca\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.044842 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-service-ca-bundle\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.044916 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac714041-979e-4c21-9610-cca061c9f3f6-tmp\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.044973 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6f6e778e-c311-4bab-b5f4-b4614e3b7795-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.045029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.044992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0092bf4b-84ef-4574-9726-4ed506b345c6-tmp\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.045082 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0092bf4b-84ef-4574-9726-4ed506b345c6-service-ca-bundle\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.045131 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.045189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph9b\" (UniqueName: \"kubernetes.io/projected/ac714041-979e-4c21-9610-cca061c9f3f6-kube-api-access-xph9b\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.045189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ac714041-979e-4c21-9610-cca061c9f3f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.045300 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-stats-auth\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.045393 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-metrics-certs\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.045510 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e808b6b-cba6-4664-9833-345b39b4f97f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.045715 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c8hjv\"" Aug 04 19:16:30.045715 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045696 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 19:16:30.045908 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.045772 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 19:16:30.046050 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.046027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f6e778e-c311-4bab-b5f4-b4614e3b7795-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.046211 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.046188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0092bf4b-84ef-4574-9726-4ed506b345c6-trusted-ca-bundle\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.046319 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.046245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0092bf4b-84ef-4574-9726-4ed506b345c6-serving-cert\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.046461 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.046322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-hub\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.064810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.064777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-vtn4d"] Aug 04 19:16:30.146712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t572g\" (UniqueName: \"kubernetes.io/projected/2451817b-9161-4742-bcbf-1cc4ebb5070e-kube-api-access-t572g\") pod \"downloads-77c974ffcf-vtn4d\" (UID: \"2451817b-9161-4742-bcbf-1cc4ebb5070e\") " pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:30.146906 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s97w\" (UniqueName: \"kubernetes.io/projected/6f6e778e-c311-4bab-b5f4-b4614e3b7795-kube-api-access-5s97w\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.146906 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcm4\" (UniqueName: \"kubernetes.io/projected/0e808b6b-cba6-4664-9833-345b39b4f97f-kube-api-access-fqcm4\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.146906 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgl7\" (UniqueName: \"kubernetes.io/projected/06bb65f1-6cae-40b3-836c-b3a3c7ca9545-kube-api-access-4jgl7\") pod \"managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z\" (UID: \"06bb65f1-6cae-40b3-836c-b3a3c7ca9545\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.146906 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-default-certificate\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0092bf4b-84ef-4574-9726-4ed506b345c6-snapshots\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.146977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhql8\" (UniqueName: \"kubernetes.io/projected/0092bf4b-84ef-4574-9726-4ed506b345c6-kube-api-access-zhql8\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-kube-api-access-nkz78\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb65f1-6cae-40b3-836c-b3a3c7ca9545-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z\" (UID: \"06bb65f1-6cae-40b3-836c-b3a3c7ca9545\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-ca\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-service-ca-bundle\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.147145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac714041-979e-4c21-9610-cca061c9f3f6-tmp\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6f6e778e-c311-4bab-b5f4-b4614e3b7795-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0092bf4b-84ef-4574-9726-4ed506b345c6-tmp\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0092bf4b-84ef-4574-9726-4ed506b345c6-service-ca-bundle\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph9b\" (UniqueName: \"kubernetes.io/projected/ac714041-979e-4c21-9610-cca061c9f3f6-kube-api-access-xph9b\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ac714041-979e-4c21-9610-cca061c9f3f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-stats-auth\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-metrics-certs\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e808b6b-cba6-4664-9833-345b39b4f97f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f6e778e-c311-4bab-b5f4-b4614e3b7795-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0092bf4b-84ef-4574-9726-4ed506b345c6-trusted-ca-bundle\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0092bf4b-84ef-4574-9726-4ed506b345c6-serving-cert\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.147632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.147586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-hub\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.148316 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.148226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0092bf4b-84ef-4574-9726-4ed506b345c6-snapshots\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.150290 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.150261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac714041-979e-4c21-9610-cca061c9f3f6-tmp\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.150661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.150634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-service-ca-bundle\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.150856 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.150833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e808b6b-cba6-4664-9833-345b39b4f97f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.151144 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.151120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0092bf4b-84ef-4574-9726-4ed506b345c6-tmp\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.151223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.151199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0092bf4b-84ef-4574-9726-4ed506b345c6-trusted-ca-bundle\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.151504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.151478 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-hub\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.151584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.151488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-metrics-certs\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.151664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.151634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-default-certificate\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.151756 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.151734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6f6e778e-c311-4bab-b5f4-b4614e3b7795-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.152045 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.152007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.152114 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.152091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0092bf4b-84ef-4574-9726-4ed506b345c6-service-ca-bundle\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.152492 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.152463 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-ca\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.153227 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.153203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-stats-auth\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.153346 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.153270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f6e778e-c311-4bab-b5f4-b4614e3b7795-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.153597 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.153577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e808b6b-cba6-4664-9833-345b39b4f97f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.153785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.153770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0092bf4b-84ef-4574-9726-4ed506b345c6-serving-cert\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.154337 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.154309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ac714041-979e-4c21-9610-cca061c9f3f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.169656 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.169620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph9b\" (UniqueName: \"kubernetes.io/projected/ac714041-979e-4c21-9610-cca061c9f3f6-kube-api-access-xph9b\") pod \"klusterlet-addon-workmgr-5bbc94ccd9-bdnlz\" (UID: \"ac714041-979e-4c21-9610-cca061c9f3f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.170559 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.170529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhql8\" (UniqueName: \"kubernetes.io/projected/0092bf4b-84ef-4574-9726-4ed506b345c6-kube-api-access-zhql8\") pod \"insights-operator-77696685b-rd8q2\" (UID: \"0092bf4b-84ef-4574-9726-4ed506b345c6\") " pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.170670 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.170542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcm4\" (UniqueName: \"kubernetes.io/projected/0e808b6b-cba6-4664-9833-345b39b4f97f-kube-api-access-fqcm4\") pod \"cluster-proxy-proxy-agent-66c88fd6b4-q77l5\" (UID: \"0e808b6b-cba6-4664-9833-345b39b4f97f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.170727 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.170709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s97w\" (UniqueName: \"kubernetes.io/projected/6f6e778e-c311-4bab-b5f4-b4614e3b7795-kube-api-access-5s97w\") pod \"cluster-monitoring-operator-6f55c4f58d-ngdn9\" (UID: \"6f6e778e-c311-4bab-b5f4-b4614e3b7795\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.173740 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.173717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/c9efa644-f790-4e4b-8ddf-a9a75d1c9719-kube-api-access-nkz78\") pod \"router-default-6dd5c49894-ktdqh\" (UID: \"c9efa644-f790-4e4b-8ddf-a9a75d1c9719\") " pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.248646 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.248540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t572g\" (UniqueName: \"kubernetes.io/projected/2451817b-9161-4742-bcbf-1cc4ebb5070e-kube-api-access-t572g\") pod \"downloads-77c974ffcf-vtn4d\" (UID: \"2451817b-9161-4742-bcbf-1cc4ebb5070e\") " pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:30.248646 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.248587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgl7\" (UniqueName: \"kubernetes.io/projected/06bb65f1-6cae-40b3-836c-b3a3c7ca9545-kube-api-access-4jgl7\") pod \"managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z\" (UID: \"06bb65f1-6cae-40b3-836c-b3a3c7ca9545\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.248895 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.248715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb65f1-6cae-40b3-836c-b3a3c7ca9545-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z\" (UID: \"06bb65f1-6cae-40b3-836c-b3a3c7ca9545\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.251198 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.251167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb65f1-6cae-40b3-836c-b3a3c7ca9545-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z\" (UID: \"06bb65f1-6cae-40b3-836c-b3a3c7ca9545\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.258560 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.258534 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgl7\" (UniqueName: \"kubernetes.io/projected/06bb65f1-6cae-40b3-836c-b3a3c7ca9545-kube-api-access-4jgl7\") pod \"managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z\" (UID: \"06bb65f1-6cae-40b3-836c-b3a3c7ca9545\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.258560 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.258547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t572g\" (UniqueName: \"kubernetes.io/projected/2451817b-9161-4742-bcbf-1cc4ebb5070e-kube-api-access-t572g\") pod \"downloads-77c974ffcf-vtn4d\" (UID: \"2451817b-9161-4742-bcbf-1cc4ebb5070e\") " pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:30.266779 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.266753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:30.274260 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.274232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-rd8q2" Aug 04 19:16:30.289291 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.289263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" Aug 04 19:16:30.295231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.295186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:30.307922 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.307895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" Aug 04 19:16:30.318077 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.317847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" Aug 04 19:16:30.369263 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.368964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:30.492673 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.492551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dd5c49894-ktdqh"] Aug 04 19:16:30.501971 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:30.501875 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9efa644_f790_4e4b_8ddf_a9a75d1c9719.slice/crio-4876fe6e363a641df8347313625a4d1b8bbdd4588f000650d153295390a19b33 WatchSource:0}: Error finding container 4876fe6e363a641df8347313625a4d1b8bbdd4588f000650d153295390a19b33: Status 404 returned error can't find the container with id 4876fe6e363a641df8347313625a4d1b8bbdd4588f000650d153295390a19b33 Aug 04 19:16:30.509211 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.509105 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-rd8q2"] Aug 04 19:16:30.549605 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.549466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz"] Aug 04 19:16:30.563230 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:30.562813 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac714041_979e_4c21_9610_cca061c9f3f6.slice/crio-5b0467bc2b79462641ff7fa5092b5e61f51304925f2365615c91f35b41339a81 WatchSource:0}: Error finding container 5b0467bc2b79462641ff7fa5092b5e61f51304925f2365615c91f35b41339a81: Status 404 returned error can't find the container with id 5b0467bc2b79462641ff7fa5092b5e61f51304925f2365615c91f35b41339a81 Aug 04 19:16:30.571675 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.571409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9"] Aug 04 19:16:30.574886 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:30.574516 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6e778e_c311_4bab_b5f4_b4614e3b7795.slice/crio-5b4cf9cdfda312b9ac4fa29a05f45a37fd785eabefe7fd4b0dccaf46e609191f WatchSource:0}: Error finding container 5b4cf9cdfda312b9ac4fa29a05f45a37fd785eabefe7fd4b0dccaf46e609191f: Status 404 returned error can't find the container with id 5b4cf9cdfda312b9ac4fa29a05f45a37fd785eabefe7fd4b0dccaf46e609191f Aug 04 19:16:30.596886 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.596814 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5"] Aug 04 19:16:30.600859 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:30.600812 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e808b6b_cba6_4664_9833_345b39b4f97f.slice/crio-04f16486aceda546ca3f3c04e2d719e05dc15b7bb71c32fa6ec66483bc0fcc20 WatchSource:0}: Error finding container 04f16486aceda546ca3f3c04e2d719e05dc15b7bb71c32fa6ec66483bc0fcc20: Status 404 returned error can't find the container with id 04f16486aceda546ca3f3c04e2d719e05dc15b7bb71c32fa6ec66483bc0fcc20 Aug 04 19:16:30.619209 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.619139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z"] Aug 04 19:16:30.625500 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:30.625466 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06bb65f1_6cae_40b3_836c_b3a3c7ca9545.slice/crio-1a444793a1c8372c801f3ec67673c7f4416ca4b57093d977087e0bda317a765e WatchSource:0}: Error finding container 1a444793a1c8372c801f3ec67673c7f4416ca4b57093d977087e0bda317a765e: Status 404 returned error can't find the container with id 1a444793a1c8372c801f3ec67673c7f4416ca4b57093d977087e0bda317a765e Aug 04 19:16:30.644451 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.644340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-vtn4d"] Aug 04 19:16:30.647896 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:30.647856 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2451817b_9161_4742_bcbf_1cc4ebb5070e.slice/crio-ab40ba3da123a1a69ef7d63708bdc3839328715ce1312632599b99990a4b5461 WatchSource:0}: Error finding container ab40ba3da123a1a69ef7d63708bdc3839328715ce1312632599b99990a4b5461: Status 404 returned error can't find the container with id ab40ba3da123a1a69ef7d63708bdc3839328715ce1312632599b99990a4b5461 Aug 04 19:16:30.962049 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.962002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" event={"ID":"0e808b6b-cba6-4664-9833-345b39b4f97f","Type":"ContainerStarted","Data":"04f16486aceda546ca3f3c04e2d719e05dc15b7bb71c32fa6ec66483bc0fcc20"} Aug 04 19:16:30.963490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.963462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" event={"ID":"ac714041-979e-4c21-9610-cca061c9f3f6","Type":"ContainerStarted","Data":"5b0467bc2b79462641ff7fa5092b5e61f51304925f2365615c91f35b41339a81"} Aug 04 19:16:30.966300 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.966211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" event={"ID":"c9efa644-f790-4e4b-8ddf-a9a75d1c9719","Type":"ContainerStarted","Data":"dc7d54debcd365a18cbe91cab5bdf0abaf0631fd12b7a77a8ca291fcaa12f30d"} Aug 04 19:16:30.966300 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.966267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" event={"ID":"c9efa644-f790-4e4b-8ddf-a9a75d1c9719","Type":"ContainerStarted","Data":"4876fe6e363a641df8347313625a4d1b8bbdd4588f000650d153295390a19b33"} Aug 04 19:16:30.969665 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.969589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-vtn4d" event={"ID":"2451817b-9161-4742-bcbf-1cc4ebb5070e","Type":"ContainerStarted","Data":"ab40ba3da123a1a69ef7d63708bdc3839328715ce1312632599b99990a4b5461"} Aug 04 19:16:30.971359 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.971304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" event={"ID":"6f6e778e-c311-4bab-b5f4-b4614e3b7795","Type":"ContainerStarted","Data":"5b4cf9cdfda312b9ac4fa29a05f45a37fd785eabefe7fd4b0dccaf46e609191f"} Aug 04 19:16:30.973390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.973363 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" event={"ID":"06bb65f1-6cae-40b3-836c-b3a3c7ca9545","Type":"ContainerStarted","Data":"1a444793a1c8372c801f3ec67673c7f4416ca4b57093d977087e0bda317a765e"} Aug 04 19:16:30.974984 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.974901 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rd8q2" event={"ID":"0092bf4b-84ef-4574-9726-4ed506b345c6","Type":"ContainerStarted","Data":"e66b63ce8bbe5c08fc7a04a32513e20704bd5ebe9b13b5717ec5bf45181e5fa2"} Aug 04 19:16:30.975231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.975210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:30.990075 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:30.990009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" podStartSLOduration=1.989991217 podStartE2EDuration="1.989991217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:30.988620341 +0000 UTC m=+67.929417112" watchObservedRunningTime="2026-08-04 19:16:30.989991217 +0000 UTC m=+67.930787964" Aug 04 19:16:31.267049 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:31.266916 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:31.270151 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:31.269911 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:32.003560 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:32.000558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnd99" event={"ID":"2a8860b7-1f1b-4269-b469-cac0bc30c406","Type":"ContainerStarted","Data":"e5c02343d5ff39f18b61afbca42975dc58c18eafbc80e800df6abbc80a7acaa0"} Aug 04 19:16:32.003560 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:32.000669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnd99" event={"ID":"2a8860b7-1f1b-4269-b469-cac0bc30c406","Type":"ContainerStarted","Data":"9a73a8cde1efe056ae8781229c5529bfeecac3b71c63e846e9b777cc536b4a4a"} Aug 04 19:16:32.003560 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:32.001809 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:32.005260 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:32.004689 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6dd5c49894-ktdqh" Aug 04 19:16:32.021188 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:32.021122 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qnd99" podStartSLOduration=67.535221263 podStartE2EDuration="1m9.021098344s" totalImagesPullingTime="1.485877081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:32.019470507 +0000 UTC m=+68.960267253" watchObservedRunningTime="2026-08-04 19:16:32.021098344 +0000 UTC m=+68.961895092" Aug 04 19:16:33.089634 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:33.089582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:33.103034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:33.099897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2709f187-9682-4546-ba3f-b8671e3f2ab7-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-x7ktf\" (UID: \"2709f187-9682-4546-ba3f-b8671e3f2ab7\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:33.277427 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:33.277197 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2f659\"" Aug 04 19:16:33.286118 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:33.286076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" Aug 04 19:16:37.865540 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:37.865506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mm924" Aug 04 19:16:40.525083 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:40.524912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf"] Aug 04 19:16:40.532130 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:40.532069 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2709f187_9682_4546_ba3f_b8671e3f2ab7.slice/crio-b2e3c93c3d0668d1e440a6c3fe4c7dc92bb3de5775eb7db529a367063f63d16e WatchSource:0}: Error finding container b2e3c93c3d0668d1e440a6c3fe4c7dc92bb3de5775eb7db529a367063f63d16e: Status 404 returned error can't find the container with id b2e3c93c3d0668d1e440a6c3fe4c7dc92bb3de5775eb7db529a367063f63d16e Aug 04 19:16:41.031030 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.030764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glb96" event={"ID":"4c10f094-a78b-4733-b541-57206eefdc6f","Type":"ContainerStarted","Data":"0bd9fa2172291105c40a6bfae091b109abf298a9a86d4416319b466f7e97a139"} Aug 04 19:16:41.032720 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.032657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-rd8q2" event={"ID":"0092bf4b-84ef-4574-9726-4ed506b345c6","Type":"ContainerStarted","Data":"807270afbaf83ceb4b4bd7b59ac018c6e0608fe7dc346d4da54cdd86769893aa"} Aug 04 19:16:41.035876 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.035388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" event={"ID":"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc","Type":"ContainerStarted","Data":"c41cc4bdc71cd623ce08dd751c2f7a4d8494ed35f216231f79aaf44c0c4e9784"} Aug 04 19:16:41.035876 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.035444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" event={"ID":"b6ce72fc-6e45-468a-a9c4-ecd6ff6c87cc","Type":"ContainerStarted","Data":"193c32461c38d334a98ff86186f824480227c7d944fa8d192a82a980a7b42ebb"} Aug 04 19:16:41.038411 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.038387 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" event={"ID":"0e808b6b-cba6-4664-9833-345b39b4f97f","Type":"ContainerStarted","Data":"ffc16bded2bb0949b4483b8c68d468ce3fed2f4b84c6d536744799dd8b29c430"} Aug 04 19:16:41.038547 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.038419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" event={"ID":"0e808b6b-cba6-4664-9833-345b39b4f97f","Type":"ContainerStarted","Data":"8bea5bcad2092b44c1c0f5262452c0a40a5be0c5cb25e4e13b00c685528fe710"} Aug 04 19:16:41.038547 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.038453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" event={"ID":"0e808b6b-cba6-4664-9833-345b39b4f97f","Type":"ContainerStarted","Data":"77e1f82defd8bb184a2ac2f48fe66943286221db4f2d005a5380e88a039ee230"} Aug 04 19:16:41.041106 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.040624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p589j" event={"ID":"6d50fb28-2b17-4db2-8465-dc86997e17ec","Type":"ContainerStarted","Data":"da35af77b631e610f401b53343d9b46f6533b1768605c71e31eb6d77b2234c0f"} Aug 04 19:16:41.041106 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.040675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p589j" event={"ID":"6d50fb28-2b17-4db2-8465-dc86997e17ec","Type":"ContainerStarted","Data":"bbaa671c4ee753d06606f9ea14de9af5641bee272d3b0b11b43104fa7cd3dd38"} Aug 04 19:16:41.041106 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.040706 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p589j" Aug 04 19:16:41.042073 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.042053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" event={"ID":"2709f187-9682-4546-ba3f-b8671e3f2ab7","Type":"ContainerStarted","Data":"b2e3c93c3d0668d1e440a6c3fe4c7dc92bb3de5775eb7db529a367063f63d16e"} Aug 04 19:16:41.043915 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.043889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" event={"ID":"6f6e778e-c311-4bab-b5f4-b4614e3b7795","Type":"ContainerStarted","Data":"1db46f58a36b46b7f93fc348ca02930337809396d2cd9e96933f8cafc5e75e74"} Aug 04 19:16:41.047467 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.047409 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-glb96" podStartSLOduration=67.605786494 podStartE2EDuration="1m18.047391884s" totalImagesPullingTime="10.44160539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:41.04595839 +0000 UTC m=+77.986755138" watchObservedRunningTime="2026-08-04 19:16:41.047391884 +0000 UTC m=+77.988188631" Aug 04 19:16:41.064033 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.063972 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p589j" podStartSLOduration=34.062758504 podStartE2EDuration="45.063951634s" totalImagesPullingTime="11.00119313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:41.063274142 +0000 UTC m=+78.004070889" watchObservedRunningTime="2026-08-04 19:16:41.063951634 +0000 UTC m=+78.004748380" Aug 04 19:16:41.086956 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.086885 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-rd8q2" podStartSLOduration=2.255738551 podStartE2EDuration="12.086861909s" totalImagesPullingTime="9.831123358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:41.084910964 +0000 UTC m=+78.025707713" watchObservedRunningTime="2026-08-04 19:16:41.086861909 +0000 UTC m=+78.027658654" Aug 04 19:16:41.105072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.105006 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-ngdn9" podStartSLOduration=2.325798954 podStartE2EDuration="12.104985525s" totalImagesPullingTime="9.779186571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:41.103175111 +0000 UTC m=+78.043971857" watchObservedRunningTime="2026-08-04 19:16:41.104985525 +0000 UTC m=+78.045782280" Aug 04 19:16:41.132971 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.132849 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66c88fd6b4-q77l5" podStartSLOduration=2.086710597 podStartE2EDuration="12.13282624s" totalImagesPullingTime="10.046115643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:41.131142057 +0000 UTC m=+78.071938804" watchObservedRunningTime="2026-08-04 19:16:41.13282624 +0000 UTC m=+78.073622984" Aug 04 19:16:41.151737 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.151640 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-9cvt2" podStartSLOduration=62.160511189 podStartE2EDuration="1m13.151617581s" totalImagesPullingTime="10.991106392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:41.150460226 +0000 UTC m=+78.091256972" watchObservedRunningTime="2026-08-04 19:16:41.151617581 +0000 UTC m=+78.092414329" Aug 04 19:16:41.277477 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.277311 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kbfzd"] Aug 04 19:16:41.286520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.286493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.290828 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.290529 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m6xxp\"" Aug 04 19:16:41.291859 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.291168 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 19:16:41.291859 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.291220 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 19:16:41.298402 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.298288 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kbfzd"] Aug 04 19:16:41.373079 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.372776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5450ac4-a4d8-498e-96ab-6a6f6424d130-crio-socket\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.373079 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.372838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.373079 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.372870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5450ac4-a4d8-498e-96ab-6a6f6424d130-data-volume\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.373079 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.372924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.373079 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.372952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltz8\" (UniqueName: \"kubernetes.io/projected/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-api-access-2ltz8\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474276 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltz8\" (UniqueName: \"kubernetes.io/projected/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-api-access-2ltz8\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5450ac4-a4d8-498e-96ab-6a6f6424d130-crio-socket\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5450ac4-a4d8-498e-96ab-6a6f6424d130-data-volume\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c5450ac4-a4d8-498e-96ab-6a6f6424d130-crio-socket\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.474758 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.474487 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:41.474758 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.474551 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.474758 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.474559 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.974538882 +0000 UTC m=+78.915335621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : configmap "kube-rbac-proxy" not found Aug 04 19:16:41.474758 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.474619 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:41.974603721 +0000 UTC m=+78.915400452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.474968 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.474851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5450ac4-a4d8-498e-96ab-6a6f6424d130-data-volume\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.484025 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.483997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltz8\" (UniqueName: \"kubernetes.io/projected/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-api-access-2ltz8\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.978988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.978950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.979409 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:41.979010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:41.979409 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.979139 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.979409 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.979147 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:41.979409 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.979223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:42.979202233 +0000 UTC m=+79.919998980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:41.979409 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:41.979286 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:42.979270645 +0000 UTC m=+79.920067370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : configmap "kube-rbac-proxy" not found Aug 04 19:16:42.050877 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:42.050794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" event={"ID":"2709f187-9682-4546-ba3f-b8671e3f2ab7","Type":"ContainerStarted","Data":"8fd616b60693e64a8d3c3eb557b7b877a49ad4c8863e75d8b091e00b16d726eb"} Aug 04 19:16:42.065850 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:42.065739 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-x7ktf" podStartSLOduration=39.641007629 podStartE2EDuration="41.065717372s" totalImagesPullingTime="1.424709743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:42.06500833 +0000 UTC m=+79.005805083" watchObservedRunningTime="2026-08-04 19:16:42.065717372 +0000 UTC m=+79.006514119" Aug 04 19:16:42.986747 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:42.986709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:42.987182 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:42.986794 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:42.987182 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:42.986833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:42.987182 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:42.986890 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:44.986854477 +0000 UTC m=+81.927651290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : configmap "kube-rbac-proxy" not found Aug 04 19:16:42.987182 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:42.987015 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:42.987182 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:42.987087 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:44.987069379 +0000 UTC m=+81.927866116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:43.052772 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:43.052732 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-x7ktf_2709f187-9682-4546-ba3f-b8671e3f2ab7/networking-console-plugin/0.log" Aug 04 19:16:43.253304 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:43.253209 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:16:43.455639 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:43.455586 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/2.log" Aug 04 19:16:44.653275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:44.653239 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p589j_6d50fb28-2b17-4db2-8465-dc86997e17ec/dns/0.log" Aug 04 19:16:44.857104 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:44.857077 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p589j_6d50fb28-2b17-4db2-8465-dc86997e17ec/kube-rbac-proxy/0.log" Aug 04 19:16:45.008590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:45.008498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:45.008590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:45.008564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:45.008820 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:45.008664 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:45.008820 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:45.008688 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:45.008820 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:45.008748 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:49.008728246 +0000 UTC m=+85.949524993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:45.008820 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:45.008777 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:49.008765413 +0000 UTC m=+85.949562138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : configmap "kube-rbac-proxy" not found Aug 04 19:16:45.452579 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:45.452546 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b6mw2_258893ce-f863-4989-bfb6-2076f423bad4/dns-node-resolver/0.log" Aug 04 19:16:46.053190 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:46.053159 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68f95bbcb8-48t5s_08cde8e9-373b-4d3c-9ec1-a970972ad1d7/registry/0.log" Aug 04 19:16:46.453263 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:46.453188 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4sdzl_d27cb5fd-9a5d-4afa-b3c1-177a83080e08/node-ca/0.log" Aug 04 19:16:47.052974 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:47.052941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd5c49894-ktdqh_c9efa644-f790-4e4b-8ddf-a9a75d1c9719/router/0.log" Aug 04 19:16:47.252978 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:47.252912 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bvf4l_65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7/serve-healthcheck-canary/0.log" Aug 04 19:16:49.048307 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:49.048260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:49.048807 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:49.048349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:49.048807 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:49.048413 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 19:16:49.048807 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:49.048439 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 19:16:49.048807 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:49.048484 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:57.048463226 +0000 UTC m=+93.989259951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : configmap "kube-rbac-proxy" not found Aug 04 19:16:49.048807 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:49.048501 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls podName:c5450ac4-a4d8-498e-96ab-6a6f6424d130 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:57.048492607 +0000 UTC m=+93.989289331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kbfzd" (UID: "c5450ac4-a4d8-498e-96ab-6a6f6424d130") : secret "insights-runtime-extractor-tls" not found Aug 04 19:16:49.130091 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:49.130052 2579 patch_prober.go:28] interesting pod/image-registry-68f95bbcb8-48t5s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 19:16:49.130274 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:49.130110 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" podUID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:16:51.056356 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.056240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p589j" Aug 04 19:16:51.101511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.101453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-vtn4d" event={"ID":"2451817b-9161-4742-bcbf-1cc4ebb5070e","Type":"ContainerStarted","Data":"1fcbb467f2ea9810bbf3996937defffd4b5d5b4b3ddf5a03bedea9d5e55c50dd"} Aug 04 19:16:51.102805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.102670 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:51.107518 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.107486 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-vtn4d" Aug 04 19:16:51.141666 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.141584 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-vtn4d" podStartSLOduration=1.580956838 podStartE2EDuration="21.141562005s" totalImagesPullingTime="19.560605167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:51.140835565 +0000 UTC m=+88.081632315" watchObservedRunningTime="2026-08-04 19:16:51.141562005 +0000 UTC m=+88.082358752" Aug 04 19:16:51.697561 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.695122 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r9fjf"] Aug 04 19:16:51.733158 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.733124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.737508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.736250 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 19:16:51.737508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.736522 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 19:16:51.737508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.736824 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 19:16:51.737508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.737046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mrvps\"" Aug 04 19:16:51.737508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.737306 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3258235-b5d4-413b-b31f-a589c2b45a25-metrics-client-ca\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-sys\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-wtmp\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhb6\" (UniqueName: \"kubernetes.io/projected/c3258235-b5d4-413b-b31f-a589c2b45a25-kube-api-access-xkhb6\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-root\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-tls\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.778590 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.778439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-textfile\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-tls\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-textfile\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3258235-b5d4-413b-b31f-a589c2b45a25-metrics-client-ca\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-sys\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-wtmp\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhb6\" (UniqueName: \"kubernetes.io/projected/c3258235-b5d4-413b-b31f-a589c2b45a25-kube-api-access-xkhb6\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-root\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.879795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-root\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:51.879910 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:16:51.879971 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-tls podName:c3258235-b5d4-413b-b31f-a589c2b45a25 nodeName:}" failed. No retries permitted until 2026-08-04 19:16:52.37994889 +0000 UTC m=+89.320745666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-tls") pod "node-exporter-r9fjf" (UID: "c3258235-b5d4-413b-b31f-a589c2b45a25") : secret "node-exporter-tls" not found Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.880297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-textfile\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.880827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3258235-b5d4-413b-b31f-a589c2b45a25-metrics-client-ca\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.881364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-wtmp\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.881876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.882440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.881945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3258235-b5d4-413b-b31f-a589c2b45a25-sys\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.891043 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.890437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:51.909016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:51.908940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhb6\" (UniqueName: \"kubernetes.io/projected/c3258235-b5d4-413b-b31f-a589c2b45a25-kube-api-access-xkhb6\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:52.014361 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:52.013391 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:16:52.386749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:52.386305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-tls\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:52.400825 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:52.400239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3258235-b5d4-413b-b31f-a589c2b45a25-node-exporter-tls\") pod \"node-exporter-r9fjf\" (UID: \"c3258235-b5d4-413b-b31f-a589c2b45a25\") " pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:52.658756 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:52.658663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9fjf" Aug 04 19:16:53.853915 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:53.853867 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f95bbcb8-48t5s"] Aug 04 19:16:54.174547 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:54.174423 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3258235_b5d4_413b_b31f_a589c2b45a25.slice/crio-3a679ff21060ecfe78e81fd0b0c6ce9be41185c3c4b6b73dbd0aac9a5d4b9ad4 WatchSource:0}: Error finding container 3a679ff21060ecfe78e81fd0b0c6ce9be41185c3c4b6b73dbd0aac9a5d4b9ad4: Status 404 returned error can't find the container with id 3a679ff21060ecfe78e81fd0b0c6ce9be41185c3c4b6b73dbd0aac9a5d4b9ad4 Aug 04 19:16:55.137625 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:55.137553 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9fjf" event={"ID":"c3258235-b5d4-413b-b31f-a589c2b45a25","Type":"ContainerStarted","Data":"3a679ff21060ecfe78e81fd0b0c6ce9be41185c3c4b6b73dbd0aac9a5d4b9ad4"} Aug 04 19:16:57.143620 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.143580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:57.144124 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.143646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:57.144417 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.144388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c5450ac4-a4d8-498e-96ab-6a6f6424d130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:57.155534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.155448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c5450ac4-a4d8-498e-96ab-6a6f6424d130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kbfzd\" (UID: \"c5450ac4-a4d8-498e-96ab-6a6f6424d130\") " pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:57.160145 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.160105 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" event={"ID":"06bb65f1-6cae-40b3-836c-b3a3c7ca9545","Type":"ContainerStarted","Data":"a015431d43932fd7252065d565376f4adcf308737c4a00ea1a5fcd230d36d7cf"} Aug 04 19:16:57.161940 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.161899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" event={"ID":"ac714041-979e-4c21-9610-cca061c9f3f6","Type":"ContainerStarted","Data":"1d9a6accc47290b2e99e206db5333469bf243bf230287abd1bd3ee6b805e1e66"} Aug 04 19:16:57.162275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.162255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:57.164691 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.164658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" Aug 04 19:16:57.176388 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.176275 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68cb4ccd44-zrd8z" podStartSLOduration=2.605326523 podStartE2EDuration="28.176259772s" totalImagesPullingTime="25.570933249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:57.175398199 +0000 UTC m=+94.116194945" watchObservedRunningTime="2026-08-04 19:16:57.176259772 +0000 UTC m=+94.117056518" Aug 04 19:16:57.199661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.199572 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bbc94ccd9-bdnlz" podStartSLOduration=2.07922038 podStartE2EDuration="28.199547937s" totalImagesPullingTime="26.120327557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:57.199037517 +0000 UTC m=+94.139834263" watchObservedRunningTime="2026-08-04 19:16:57.199547937 +0000 UTC m=+94.140344682" Aug 04 19:16:57.210343 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.210286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kbfzd" Aug 04 19:16:57.409223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:57.409122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kbfzd"] Aug 04 19:16:57.568176 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:16:57.568136 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5450ac4_a4d8_498e_96ab_6a6f6424d130.slice/crio-0590dfcdfd00845c893d39454b5eec38346ff847508d89a1de6a5b94120e6fb3 WatchSource:0}: Error finding container 0590dfcdfd00845c893d39454b5eec38346ff847508d89a1de6a5b94120e6fb3: Status 404 returned error can't find the container with id 0590dfcdfd00845c893d39454b5eec38346ff847508d89a1de6a5b94120e6fb3 Aug 04 19:16:58.168748 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:58.168708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbfzd" event={"ID":"c5450ac4-a4d8-498e-96ab-6a6f6424d130","Type":"ContainerStarted","Data":"ab6ea72ccae1bd28a9d95eaa96b9ead394cf5d79bcfd9c0ac82ccb4cf7491383"} Aug 04 19:16:58.169283 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:58.168787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbfzd" event={"ID":"c5450ac4-a4d8-498e-96ab-6a6f6424d130","Type":"ContainerStarted","Data":"0590dfcdfd00845c893d39454b5eec38346ff847508d89a1de6a5b94120e6fb3"} Aug 04 19:16:58.170753 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:58.170681 2579 generic.go:356] "Generic (PLEG): container finished" podID="c3258235-b5d4-413b-b31f-a589c2b45a25" containerID="3be32287a0c93967f997f2a5bc50926799fae8d8504ef1cce92f165b852255cb" exitCode=0 Aug 04 19:16:58.170896 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:58.170778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9fjf" event={"ID":"c3258235-b5d4-413b-b31f-a589c2b45a25","Type":"ContainerDied","Data":"3be32287a0c93967f997f2a5bc50926799fae8d8504ef1cce92f165b852255cb"} Aug 04 19:16:59.180388 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:59.180261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbfzd" event={"ID":"c5450ac4-a4d8-498e-96ab-6a6f6424d130","Type":"ContainerStarted","Data":"c65646b9ff223db0b87316d47e2f749d74cbff7092a9e9b1c9a4fe47f7a7efc9"} Aug 04 19:16:59.182394 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:59.182363 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9fjf" event={"ID":"c3258235-b5d4-413b-b31f-a589c2b45a25","Type":"ContainerStarted","Data":"96bafd46aa3638975cd466de7ffa0440a2076049f3c10363e3dbe32bfff80486"} Aug 04 19:16:59.182536 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:59.182415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9fjf" event={"ID":"c3258235-b5d4-413b-b31f-a589c2b45a25","Type":"ContainerStarted","Data":"d0a5f0fd8cc590bb9310f8423dc3b23d0d7d748658e0a2b3f881205e164b7fc4"} Aug 04 19:16:59.202796 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:16:59.202740 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r9fjf" podStartSLOduration=4.681666224 podStartE2EDuration="8.202721346s" totalImagesPullingTime="3.416624972s" totalInitContainerRuntime="104.43015ms" isStatefulPod=true podCreationTimestamp="2026-08-04 19:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:16:59.201002082 +0000 UTC m=+96.141798840" watchObservedRunningTime="2026-08-04 19:16:59.202721346 +0000 UTC m=+96.143518089" Aug 04 19:17:02.197990 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:02.197954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kbfzd" event={"ID":"c5450ac4-a4d8-498e-96ab-6a6f6424d130","Type":"ContainerStarted","Data":"a0228bcae604c3c5e7ab23fdc6d39749f62b5c64ad46a725180be77b2e94c474"} Aug 04 19:17:02.217297 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:02.217244 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kbfzd" podStartSLOduration=17.992175055 podStartE2EDuration="21.217225957s" totalImagesPullingTime="3.225050902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:16:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:17:02.216036274 +0000 UTC m=+99.156833020" watchObservedRunningTime="2026-08-04 19:17:02.217225957 +0000 UTC m=+99.158022703" Aug 04 19:17:17.257941 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:17.257851 2579 generic.go:356] "Generic (PLEG): container finished" podID="b279966b-352c-4362-9465-2bc0b9da85e5" containerID="6a36ea644e800d6b160fc61412cf45a95233915c6511aab77818ebafd47d8971" exitCode=0 Aug 04 19:17:17.257941 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:17.257928 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" event={"ID":"b279966b-352c-4362-9465-2bc0b9da85e5","Type":"ContainerDied","Data":"6a36ea644e800d6b160fc61412cf45a95233915c6511aab77818ebafd47d8971"} Aug 04 19:17:17.258458 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:17.258255 2579 scope.go:117] "RemoveContainer" containerID="6a36ea644e800d6b160fc61412cf45a95233915c6511aab77818ebafd47d8971" Aug 04 19:17:18.263880 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:18.263843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-fg8t9" event={"ID":"b279966b-352c-4362-9465-2bc0b9da85e5","Type":"ContainerStarted","Data":"015eb8f6d3e216443f89e9747ac56ac2950078dd8f59ca4f739f464a2548e298"} Aug 04 19:17:18.880606 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:18.880558 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" podUID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" containerName="registry" containerID="cri-o://f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109" gracePeriod=30 Aug 04 19:17:19.163983 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.163947 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:17:19.268543 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.268508 2579 generic.go:356] "Generic (PLEG): container finished" podID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" containerID="f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109" exitCode=0 Aug 04 19:17:19.268543 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.268548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" event={"ID":"08cde8e9-373b-4d3c-9ec1-a970972ad1d7","Type":"ContainerDied","Data":"f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109"} Aug 04 19:17:19.269000 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.268555 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" Aug 04 19:17:19.269000 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.268587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f95bbcb8-48t5s" event={"ID":"08cde8e9-373b-4d3c-9ec1-a970972ad1d7","Type":"ContainerDied","Data":"059e92b4064fe8174495e24eeff70fd9ef7b3080e7420f7794259b2b11b6e19a"} Aug 04 19:17:19.269000 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.268607 2579 scope.go:117] "RemoveContainer" containerID="f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109" Aug 04 19:17:19.276741 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.276720 2579 scope.go:117] "RemoveContainer" containerID="f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109" Aug 04 19:17:19.277014 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:17:19.276993 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109\": container with ID starting with f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109 not found: ID does not exist" containerID="f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109" Aug 04 19:17:19.277060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.277024 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109"} err="failed to get container status \"f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109\": rpc error: code = NotFound desc = could not find container \"f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109\": container with ID starting with f59f746f8eb030a0c5dcae4bdf05b24031adbe34c08f12c091002afefaea7109 not found: ID does not exist" Aug 04 19:17:19.337043 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337001 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-certificates\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337052 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-trusted-ca\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337100 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-installation-pull-secrets\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337134 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-ca-trust-extracted\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337188 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-image-registry-private-configuration\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337466 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49fb8\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-kube-api-access-49fb8\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337466 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337284 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337466 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-bound-sa-token\") pod \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\" (UID: \"08cde8e9-373b-4d3c-9ec1-a970972ad1d7\") " Aug 04 19:17:19.337609 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:19.337675 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337620 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-certificates\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.337675 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.337633 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:17:19.339899 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.339832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:17:19.339899 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.339832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-kube-api-access-49fb8" (OuterVolumeSpecName: "kube-api-access-49fb8") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "kube-api-access-49fb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:17:19.340092 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.339963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:17:19.340092 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.340069 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:17:19.340165 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.340117 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:17:19.345822 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.345793 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "08cde8e9-373b-4d3c-9ec1-a970972ad1d7" (UID: "08cde8e9-373b-4d3c-9ec1-a970972ad1d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438736 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-image-registry-private-configuration\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438782 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49fb8\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-kube-api-access-49fb8\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438795 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-registry-tls\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438809 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-bound-sa-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438817 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-trusted-ca\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438825 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-installation-pull-secrets\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.438835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.438834 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08cde8e9-373b-4d3c-9ec1-a970972ad1d7-ca-trust-extracted\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:17:19.591136 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.591105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f95bbcb8-48t5s"] Aug 04 19:17:19.595211 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.595183 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68f95bbcb8-48t5s"] Aug 04 19:17:19.609140 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:19.609105 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" path="/var/lib/kubelet/pods/08cde8e9-373b-4d3c-9ec1-a970972ad1d7/volumes" Aug 04 19:17:23.285466 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:23.285429 2579 generic.go:356] "Generic (PLEG): container finished" podID="526fc512-89e6-4817-8d17-b561d2c1e7c9" containerID="5631312a9754e7d0b0aced41d78e6704209283486257a31c643885ae3e9e4edc" exitCode=0 Aug 04 19:17:23.285917 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:23.285492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" event={"ID":"526fc512-89e6-4817-8d17-b561d2c1e7c9","Type":"ContainerDied","Data":"5631312a9754e7d0b0aced41d78e6704209283486257a31c643885ae3e9e4edc"} Aug 04 19:17:23.285917 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:23.285821 2579 scope.go:117] "RemoveContainer" containerID="5631312a9754e7d0b0aced41d78e6704209283486257a31c643885ae3e9e4edc" Aug 04 19:17:24.291014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:17:24.290976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-nlhhn" event={"ID":"526fc512-89e6-4817-8d17-b561d2c1e7c9","Type":"ContainerStarted","Data":"826969142c9cc7f6367d67630d55af7ff38a0c20c04ef95b1cc6f2b39258e325"} Aug 04 19:20:22.207741 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.207704 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw"] Aug 04 19:20:22.208171 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.207995 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" containerName="registry" Aug 04 19:20:22.208171 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.208020 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" containerName="registry" Aug 04 19:20:22.208171 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.208090 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cde8e9-373b-4d3c-9ec1-a970972ad1d7" containerName="registry" Aug 04 19:20:22.209895 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.209877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.215090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.215065 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 19:20:22.215446 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.215361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 19:20:22.215849 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.215832 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 19:20:22.215944 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.215867 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9bp4b\"" Aug 04 19:20:22.216192 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.216176 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 19:20:22.232693 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.232657 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw"] Aug 04 19:20:22.314805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.314762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1de709ab-960b-4bcc-a7b9-58a687572868-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.314985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.314834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tnj\" (UniqueName: \"kubernetes.io/projected/1de709ab-960b-4bcc-a7b9-58a687572868-kube-api-access-t9tnj\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.314985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.314875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de709ab-960b-4bcc-a7b9-58a687572868-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.314985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.314901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1de709ab-960b-4bcc-a7b9-58a687572868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.314985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.314939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1de709ab-960b-4bcc-a7b9-58a687572868-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.415931 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.415894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de709ab-960b-4bcc-a7b9-58a687572868-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.415931 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.415935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1de709ab-960b-4bcc-a7b9-58a687572868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.416134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.415956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1de709ab-960b-4bcc-a7b9-58a687572868-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.416134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.416012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1de709ab-960b-4bcc-a7b9-58a687572868-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.416134 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.416071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tnj\" (UniqueName: \"kubernetes.io/projected/1de709ab-960b-4bcc-a7b9-58a687572868-kube-api-access-t9tnj\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.416495 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.416465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1de709ab-960b-4bcc-a7b9-58a687572868-manifests\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.418301 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.418277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de709ab-960b-4bcc-a7b9-58a687572868-tmp\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.418418 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.418398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1de709ab-960b-4bcc-a7b9-58a687572868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.418634 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.418610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1de709ab-960b-4bcc-a7b9-58a687572868-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.427087 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.427064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tnj\" (UniqueName: \"kubernetes.io/projected/1de709ab-960b-4bcc-a7b9-58a687572868-kube-api-access-t9tnj\") pod \"opendatahub-operator-controller-manager-55bb79d9bb-ljzlw\" (UID: \"1de709ab-960b-4bcc-a7b9-58a687572868\") " pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.520520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.520432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:22.660084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.660060 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw"] Aug 04 19:20:22.662526 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:20:22.662498 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de709ab_960b_4bcc_a7b9_58a687572868.slice/crio-321e2321eae543291c8bf3cb9eb82517106a8a64772ee08d527f9a9f7ea898d4 WatchSource:0}: Error finding container 321e2321eae543291c8bf3cb9eb82517106a8a64772ee08d527f9a9f7ea898d4: Status 404 returned error can't find the container with id 321e2321eae543291c8bf3cb9eb82517106a8a64772ee08d527f9a9f7ea898d4 Aug 04 19:20:22.972563 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:22.972524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" event={"ID":"1de709ab-960b-4bcc-a7b9-58a687572868","Type":"ContainerStarted","Data":"321e2321eae543291c8bf3cb9eb82517106a8a64772ee08d527f9a9f7ea898d4"} Aug 04 19:20:23.484488 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:23.484456 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:20:23.498199 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:23.498168 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:20:23.505257 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:23.505229 2579 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 19:20:25.989777 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:25.989679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" event={"ID":"1de709ab-960b-4bcc-a7b9-58a687572868","Type":"ContainerStarted","Data":"dc774f2bc812f88d6fff2c409a74ffb5f718028b98ad4def87b0585c3d465db7"} Aug 04 19:20:26.995288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:26.995247 2579 generic.go:356] "Generic (PLEG): container finished" podID="1de709ab-960b-4bcc-a7b9-58a687572868" containerID="dc774f2bc812f88d6fff2c409a74ffb5f718028b98ad4def87b0585c3d465db7" exitCode=0 Aug 04 19:20:26.995860 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:26.995344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" event={"ID":"1de709ab-960b-4bcc-a7b9-58a687572868","Type":"ContainerDied","Data":"dc774f2bc812f88d6fff2c409a74ffb5f718028b98ad4def87b0585c3d465db7"} Aug 04 19:20:26.995860 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:26.995841 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:20:28.001236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:28.001198 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" event={"ID":"1de709ab-960b-4bcc-a7b9-58a687572868","Type":"ContainerStarted","Data":"ae2a7cd64a781e8061da64df8b5c21815021149b8b65608d0555cc61d2595b1a"} Aug 04 19:20:28.001666 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:28.001357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:28.021294 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:28.021246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" podStartSLOduration=1.596102602 podStartE2EDuration="6.021230395s" totalImagesPullingTime="3.376183918s" totalInitContainerRuntime="1.048943875s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:28.019942404 +0000 UTC m=+304.960739150" watchObservedRunningTime="2026-08-04 19:20:28.021230395 +0000 UTC m=+304.962027140" Aug 04 19:20:39.007544 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:39.007511 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55bb79d9bb-ljzlw" Aug 04 19:20:49.324359 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.324310 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d56c454c-45dps"] Aug 04 19:20:49.326621 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.326604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.329187 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.329164 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 19:20:49.330050 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.330027 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 19:20:49.330179 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.330111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xf98l\"" Aug 04 19:20:49.330179 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.330114 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 19:20:49.330343 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.330120 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 19:20:49.330467 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.330449 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 19:20:49.345541 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.345465 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d56c454c-45dps"] Aug 04 19:20:49.447475 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.447438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532253aa-022c-4fff-930a-1125f88e260c-cert\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.447649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.447502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/532253aa-022c-4fff-930a-1125f88e260c-manager-config\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.447649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.447525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52h8t\" (UniqueName: \"kubernetes.io/projected/532253aa-022c-4fff-930a-1125f88e260c-kube-api-access-52h8t\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.447649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.447573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/532253aa-022c-4fff-930a-1125f88e260c-metrics-cert\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.548060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.548017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/532253aa-022c-4fff-930a-1125f88e260c-manager-config\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.548060 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.548065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52h8t\" (UniqueName: \"kubernetes.io/projected/532253aa-022c-4fff-930a-1125f88e260c-kube-api-access-52h8t\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.548275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.548112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/532253aa-022c-4fff-930a-1125f88e260c-metrics-cert\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.548275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.548161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532253aa-022c-4fff-930a-1125f88e260c-cert\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.548691 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.548663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/532253aa-022c-4fff-930a-1125f88e260c-manager-config\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.550740 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.550718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/532253aa-022c-4fff-930a-1125f88e260c-cert\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.550740 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.550733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/532253aa-022c-4fff-930a-1125f88e260c-metrics-cert\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.577997 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.577918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52h8t\" (UniqueName: \"kubernetes.io/projected/532253aa-022c-4fff-930a-1125f88e260c-kube-api-access-52h8t\") pod \"lws-controller-manager-59d56c454c-45dps\" (UID: \"532253aa-022c-4fff-930a-1125f88e260c\") " pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.639055 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.639009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:49.795740 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:20:49.795708 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532253aa_022c_4fff_930a_1125f88e260c.slice/crio-4f00eda995f4b89f9752168cd5538d922a5d65c7a2aba57c680a989fcc972fe3 WatchSource:0}: Error finding container 4f00eda995f4b89f9752168cd5538d922a5d65c7a2aba57c680a989fcc972fe3: Status 404 returned error can't find the container with id 4f00eda995f4b89f9752168cd5538d922a5d65c7a2aba57c680a989fcc972fe3 Aug 04 19:20:49.802646 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:49.802615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d56c454c-45dps"] Aug 04 19:20:50.099429 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:50.099393 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" event={"ID":"532253aa-022c-4fff-930a-1125f88e260c","Type":"ContainerStarted","Data":"4f00eda995f4b89f9752168cd5538d922a5d65c7a2aba57c680a989fcc972fe3"} Aug 04 19:20:54.121462 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:54.121425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" event={"ID":"532253aa-022c-4fff-930a-1125f88e260c","Type":"ContainerStarted","Data":"c1102d9850e858efcf0f175202a73cb3dedf2a659f5ce828f46db6809debcf99"} Aug 04 19:20:54.121881 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:54.121543 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:20:54.181677 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:54.181619 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" podStartSLOduration=1.682089554 podStartE2EDuration="5.181598694s" totalImagesPullingTime="3.49950914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:20:54.174970451 +0000 UTC m=+331.115767226" watchObservedRunningTime="2026-08-04 19:20:54.181598694 +0000 UTC m=+331.122395440" Aug 04 19:20:57.915429 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:57.915398 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4"] Aug 04 19:20:57.917889 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:57.917873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:57.920411 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:57.920383 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bnzld\"" Aug 04 19:20:57.921756 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:57.921731 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 19:20:57.938843 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:57.938812 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4"] Aug 04 19:20:58.034502 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0bff41-45c7-4708-a703-b88f946d6da3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034502 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7clt\" (UniqueName: \"kubernetes.io/projected/0f0bff41-45c7-4708-a703-b88f946d6da3-kube-api-access-d7clt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034752 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034752 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034752 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034752 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034713 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034752 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034981 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.034981 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.034878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135291 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0bff41-45c7-4708-a703-b88f946d6da3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7clt\" (UniqueName: \"kubernetes.io/projected/0f0bff41-45c7-4708-a703-b88f946d6da3-kube-api-access-d7clt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.135867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.135812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.136176 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.136149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0bff41-45c7-4708-a703-b88f946d6da3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.136365 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.136199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.136467 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.136289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.138576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.138549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.138720 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.138704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.151893 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.151851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0bff41-45c7-4708-a703-b88f946d6da3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.152503 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.152477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7clt\" (UniqueName: \"kubernetes.io/projected/0f0bff41-45c7-4708-a703-b88f946d6da3-kube-api-access-d7clt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4\" (UID: \"0f0bff41-45c7-4708-a703-b88f946d6da3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.228403 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.228263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:20:58.396743 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:20:58.396711 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0bff41_45c7_4708_a703_b88f946d6da3.slice/crio-76f06db47be52ac669a12d31c7bda8fcf24a1bd8bb4b977f232742200201ee4a WatchSource:0}: Error finding container 76f06db47be52ac669a12d31c7bda8fcf24a1bd8bb4b977f232742200201ee4a: Status 404 returned error can't find the container with id 76f06db47be52ac669a12d31c7bda8fcf24a1bd8bb4b977f232742200201ee4a Aug 04 19:20:58.399851 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:58.399825 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4"] Aug 04 19:20:59.147668 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:20:59.147633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" event={"ID":"0f0bff41-45c7-4708-a703-b88f946d6da3","Type":"ContainerStarted","Data":"76f06db47be52ac669a12d31c7bda8fcf24a1bd8bb4b977f232742200201ee4a"} Aug 04 19:21:02.574197 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:02.574154 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:21:02.574470 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:02.574253 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:21:02.574470 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:02.574292 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:21:03.168947 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:03.168915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" event={"ID":"0f0bff41-45c7-4708-a703-b88f946d6da3","Type":"ContainerStarted","Data":"ebcbab3971c3f7dc64716c5dd8af3071de6dcd045ee505d516945ecd037be947"} Aug 04 19:21:03.195817 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:03.195760 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" podStartSLOduration=2.020503153 podStartE2EDuration="6.195742106s" totalImagesPullingTime="4.175238953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:21:03.192999961 +0000 UTC m=+340.133796706" watchObservedRunningTime="2026-08-04 19:21:03.195742106 +0000 UTC m=+340.136538851" Aug 04 19:21:03.229213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:03.229166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:21:03.234493 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:03.234460 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:21:04.174888 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:04.174851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:21:04.175961 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:04.175935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4" Aug 04 19:21:05.128494 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:21:05.128450 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59d56c454c-45dps" Aug 04 19:22:02.821569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.821524 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-68kqs"] Aug 04 19:22:02.826973 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.826945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:02.829420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.829390 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 19:22:02.829583 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.829421 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 19:22:02.829741 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.829725 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-c9nqv\"" Aug 04 19:22:02.830370 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.830352 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 19:22:02.836078 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.836058 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-68kqs"] Aug 04 19:22:02.885919 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.885880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjk9\" (UniqueName: \"kubernetes.io/projected/da8d7e93-d70b-401f-a5ba-d302696cd9ef-kube-api-access-khjk9\") pod \"dns-operator-controller-manager-5985564776-68kqs\" (UID: \"da8d7e93-d70b-401f-a5ba-d302696cd9ef\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:02.987063 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:02.987023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjk9\" (UniqueName: \"kubernetes.io/projected/da8d7e93-d70b-401f-a5ba-d302696cd9ef-kube-api-access-khjk9\") pod \"dns-operator-controller-manager-5985564776-68kqs\" (UID: \"da8d7e93-d70b-401f-a5ba-d302696cd9ef\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:03.000785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:03.000745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjk9\" (UniqueName: \"kubernetes.io/projected/da8d7e93-d70b-401f-a5ba-d302696cd9ef-kube-api-access-khjk9\") pod \"dns-operator-controller-manager-5985564776-68kqs\" (UID: \"da8d7e93-d70b-401f-a5ba-d302696cd9ef\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:03.137788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:03.137752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:03.292610 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:03.292577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-68kqs"] Aug 04 19:22:03.295200 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:22:03.295167 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8d7e93_d70b_401f_a5ba_d302696cd9ef.slice/crio-c16069cddff8ed2582d3bcf16bf837e64cbaf3bbb6294e8f6f8cf387fcc50c90 WatchSource:0}: Error finding container c16069cddff8ed2582d3bcf16bf837e64cbaf3bbb6294e8f6f8cf387fcc50c90: Status 404 returned error can't find the container with id c16069cddff8ed2582d3bcf16bf837e64cbaf3bbb6294e8f6f8cf387fcc50c90 Aug 04 19:22:03.432822 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:03.432739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" event={"ID":"da8d7e93-d70b-401f-a5ba-d302696cd9ef","Type":"ContainerStarted","Data":"c16069cddff8ed2582d3bcf16bf837e64cbaf3bbb6294e8f6f8cf387fcc50c90"} Aug 04 19:22:06.449971 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:06.449934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" event={"ID":"da8d7e93-d70b-401f-a5ba-d302696cd9ef","Type":"ContainerStarted","Data":"7058b4c937fc3013475c6970fcee49102ca049332d99ce8575705578093a12f1"} Aug 04 19:22:06.450370 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:06.450066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:06.470372 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:06.470299 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" podStartSLOduration=1.524043753 podStartE2EDuration="4.470284159s" totalImagesPullingTime="2.946240406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:06.468252379 +0000 UTC m=+403.409049107" watchObservedRunningTime="2026-08-04 19:22:06.470284159 +0000 UTC m=+403.411080904" Aug 04 19:22:07.514773 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.514739 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8"] Aug 04 19:22:07.518110 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.518087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.520950 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.520930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-846sv\"" Aug 04 19:22:07.531825 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.531796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8"] Aug 04 19:22:07.623185 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.623148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mrw\" (UniqueName: \"kubernetes.io/projected/154c6634-62cf-403e-bf93-7911e86f1f93-kube-api-access-b6mrw\") pod \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.623436 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.623250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/154c6634-62cf-403e-bf93-7911e86f1f93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.724618 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.724575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/154c6634-62cf-403e-bf93-7911e86f1f93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.724788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.724748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mrw\" (UniqueName: \"kubernetes.io/projected/154c6634-62cf-403e-bf93-7911e86f1f93-kube-api-access-b6mrw\") pod \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.724991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.724973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/154c6634-62cf-403e-bf93-7911e86f1f93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.733810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.733784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mrw\" (UniqueName: \"kubernetes.io/projected/154c6634-62cf-403e-bf93-7911e86f1f93-kube-api-access-b6mrw\") pod \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.828599 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.828558 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:07.987369 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:07.987342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8"] Aug 04 19:22:07.989835 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:22:07.989806 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154c6634_62cf_403e_bf93_7911e86f1f93.slice/crio-f1b37560204675a816eddc6a4a194267d5124a3cd8e9ad41617be6024c10c2b4 WatchSource:0}: Error finding container f1b37560204675a816eddc6a4a194267d5124a3cd8e9ad41617be6024c10c2b4: Status 404 returned error can't find the container with id f1b37560204675a816eddc6a4a194267d5124a3cd8e9ad41617be6024c10c2b4 Aug 04 19:22:08.459352 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:08.459291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" event={"ID":"154c6634-62cf-403e-bf93-7911e86f1f93","Type":"ContainerStarted","Data":"f1b37560204675a816eddc6a4a194267d5124a3cd8e9ad41617be6024c10c2b4"} Aug 04 19:22:12.481078 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:12.481035 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" event={"ID":"154c6634-62cf-403e-bf93-7911e86f1f93","Type":"ContainerStarted","Data":"2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f"} Aug 04 19:22:12.481546 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:12.481131 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:12.512399 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:12.512280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" podStartSLOduration=1.255781448 podStartE2EDuration="5.512262734s" totalImagesPullingTime="4.256481286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:12.507239224 +0000 UTC m=+409.448035973" watchObservedRunningTime="2026-08-04 19:22:12.512262734 +0000 UTC m=+409.453059559" Aug 04 19:22:17.456130 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:17.456095 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-68kqs" Aug 04 19:22:23.487826 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:23.487792 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.214442 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.214409 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8"] Aug 04 19:22:25.214870 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.214597 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" containerName="manager" containerID="cri-o://2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f" gracePeriod=2 Aug 04 19:22:25.223344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.223297 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8"] Aug 04 19:22:25.271105 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.271067 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h"] Aug 04 19:22:25.271480 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.271438 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" containerName="manager" Aug 04 19:22:25.271480 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.271458 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" containerName="manager" Aug 04 19:22:25.271810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.271558 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" containerName="manager" Aug 04 19:22:25.274859 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.274826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.275014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.274974 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz"] Aug 04 19:22:25.278940 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.278911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.280259 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:22:25.280209 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" is forbidden: User \"system:node:ip-10-0-143-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-62.ec2.internal' and this object" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.296845 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.296545 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h"] Aug 04 19:22:25.300288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.300260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz"] Aug 04 19:22:25.363211 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:22:25.363162 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" is forbidden: User \"system:node:ip-10-0-143-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-62.ec2.internal' and this object" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.377739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.377704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62zq\" (UniqueName: \"kubernetes.io/projected/f9718008-dd4b-4bb5-b005-395a18daf787-kube-api-access-n62zq\") pod \"kuadrant-operator-controller-manager-58fc56757c-s5b4h\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.377900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.377761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-6nqdz\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.377900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.377822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9718008-dd4b-4bb5-b005-395a18daf787-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s5b4h\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.377900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.377865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztppd\" (UniqueName: \"kubernetes.io/projected/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-kube-api-access-ztppd\") pod \"kuadrant-operator-controller-manager-68cf589589-6nqdz\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.471482 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.471410 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.473657 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:22:25.473626 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" is forbidden: User \"system:node:ip-10-0-143-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-62.ec2.internal' and this object" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.479085 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.479050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-6nqdz\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.479234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.479111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9718008-dd4b-4bb5-b005-395a18daf787-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s5b4h\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.479234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.479165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztppd\" (UniqueName: \"kubernetes.io/projected/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-kube-api-access-ztppd\") pod \"kuadrant-operator-controller-manager-68cf589589-6nqdz\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.479234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.479224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62zq\" (UniqueName: \"kubernetes.io/projected/f9718008-dd4b-4bb5-b005-395a18daf787-kube-api-access-n62zq\") pod \"kuadrant-operator-controller-manager-58fc56757c-s5b4h\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.479485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.479462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-6nqdz\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.479564 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.479542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9718008-dd4b-4bb5-b005-395a18daf787-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-s5b4h\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.491856 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.491825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztppd\" (UniqueName: \"kubernetes.io/projected/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-kube-api-access-ztppd\") pod \"kuadrant-operator-controller-manager-68cf589589-6nqdz\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.491964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.491934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62zq\" (UniqueName: \"kubernetes.io/projected/f9718008-dd4b-4bb5-b005-395a18daf787-kube-api-access-n62zq\") pod \"kuadrant-operator-controller-manager-58fc56757c-s5b4h\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.542654 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.542619 2579 generic.go:356] "Generic (PLEG): container finished" podID="154c6634-62cf-403e-bf93-7911e86f1f93" containerID="2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f" exitCode=0 Aug 04 19:22:25.542821 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.542663 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.542821 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.542690 2579 scope.go:117] "RemoveContainer" containerID="2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f" Aug 04 19:22:25.544928 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:22:25.544900 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-mzqc8\" is forbidden: User \"system:node:ip-10-0-143-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-62.ec2.internal' and this object" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-mzqc8" Aug 04 19:22:25.551379 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.551355 2579 scope.go:117] "RemoveContainer" containerID="2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f" Aug 04 19:22:25.551688 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:22:25.551664 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f\": container with ID starting with 2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f not found: ID does not exist" containerID="2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f" Aug 04 19:22:25.551754 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.551699 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f"} err="failed to get container status \"2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f\": rpc error: code = NotFound desc = could not find container \"2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f\": container with ID starting with 2be3bfeb3cd610932cc00cb0643b01f3b660a2933ed0152bae1262bd157ec81f not found: ID does not exist" Aug 04 19:22:25.580076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.580036 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6mrw\" (UniqueName: \"kubernetes.io/projected/154c6634-62cf-403e-bf93-7911e86f1f93-kube-api-access-b6mrw\") pod \"154c6634-62cf-403e-bf93-7911e86f1f93\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " Aug 04 19:22:25.580248 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.580090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/154c6634-62cf-403e-bf93-7911e86f1f93-extensions-socket-volume\") pod \"154c6634-62cf-403e-bf93-7911e86f1f93\" (UID: \"154c6634-62cf-403e-bf93-7911e86f1f93\") " Aug 04 19:22:25.580550 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.580516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154c6634-62cf-403e-bf93-7911e86f1f93-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "154c6634-62cf-403e-bf93-7911e86f1f93" (UID: "154c6634-62cf-403e-bf93-7911e86f1f93"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:22:25.582321 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.582291 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154c6634-62cf-403e-bf93-7911e86f1f93-kube-api-access-b6mrw" (OuterVolumeSpecName: "kube-api-access-b6mrw") pod "154c6634-62cf-403e-bf93-7911e86f1f93" (UID: "154c6634-62cf-403e-bf93-7911e86f1f93"). InnerVolumeSpecName "kube-api-access-b6mrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:22:25.603495 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.603457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:25.610683 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.610649 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="154c6634-62cf-403e-bf93-7911e86f1f93" path="/var/lib/kubelet/pods/154c6634-62cf-403e-bf93-7911e86f1f93/volumes" Aug 04 19:22:25.611125 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.611100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:25.681302 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.681259 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6mrw\" (UniqueName: \"kubernetes.io/projected/154c6634-62cf-403e-bf93-7911e86f1f93-kube-api-access-b6mrw\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:22:25.681302 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.681299 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/154c6634-62cf-403e-bf93-7911e86f1f93-extensions-socket-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:22:25.778495 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.778472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h"] Aug 04 19:22:25.780867 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:22:25.780839 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9718008_dd4b_4bb5_b005_395a18daf787.slice/crio-6194dc60ecf22adff0f7361ccaa6f8aa7b95779c286212233285f0d570623cb4 WatchSource:0}: Error finding container 6194dc60ecf22adff0f7361ccaa6f8aa7b95779c286212233285f0d570623cb4: Status 404 returned error can't find the container with id 6194dc60ecf22adff0f7361ccaa6f8aa7b95779c286212233285f0d570623cb4 Aug 04 19:22:25.792758 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:25.792721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz"] Aug 04 19:22:25.798151 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:22:25.798113 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de813bd_c7f2_46d3_a22c_31e79fa9ce97.slice/crio-fa051e794efc9a8a1404ac2e1f2d0e1a44504fb4dbd23d41a3d28625af5c4f9a WatchSource:0}: Error finding container fa051e794efc9a8a1404ac2e1f2d0e1a44504fb4dbd23d41a3d28625af5c4f9a: Status 404 returned error can't find the container with id fa051e794efc9a8a1404ac2e1f2d0e1a44504fb4dbd23d41a3d28625af5c4f9a Aug 04 19:22:26.550440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.550401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" event={"ID":"f9718008-dd4b-4bb5-b005-395a18daf787","Type":"ContainerStarted","Data":"6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c"} Aug 04 19:22:26.550868 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.550496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:26.550868 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.550521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" event={"ID":"f9718008-dd4b-4bb5-b005-395a18daf787","Type":"ContainerStarted","Data":"6194dc60ecf22adff0f7361ccaa6f8aa7b95779c286212233285f0d570623cb4"} Aug 04 19:22:26.552092 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.552061 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" event={"ID":"1de813bd-c7f2-46d3-a22c-31e79fa9ce97","Type":"ContainerStarted","Data":"acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b"} Aug 04 19:22:26.552221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.552110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" event={"ID":"1de813bd-c7f2-46d3-a22c-31e79fa9ce97","Type":"ContainerStarted","Data":"fa051e794efc9a8a1404ac2e1f2d0e1a44504fb4dbd23d41a3d28625af5c4f9a"} Aug 04 19:22:26.552221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.552129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:26.575213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.575160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" podStartSLOduration=1.57514219 podStartE2EDuration="1.57514219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:26.573039297 +0000 UTC m=+423.513836042" watchObservedRunningTime="2026-08-04 19:22:26.57514219 +0000 UTC m=+423.515938935" Aug 04 19:22:26.602193 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:26.602134 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" podStartSLOduration=1.602115581 podStartE2EDuration="1.602115581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:26.601098651 +0000 UTC m=+423.541895397" watchObservedRunningTime="2026-08-04 19:22:26.602115581 +0000 UTC m=+423.542912327" Aug 04 19:22:37.560519 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.560486 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:22:37.561022 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.560556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:37.651340 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.651287 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h"] Aug 04 19:22:37.651551 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.651511 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" podUID="f9718008-dd4b-4bb5-b005-395a18daf787" containerName="manager" containerID="cri-o://6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c" gracePeriod=10 Aug 04 19:22:37.899664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.899631 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:37.988504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.988471 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n62zq\" (UniqueName: \"kubernetes.io/projected/f9718008-dd4b-4bb5-b005-395a18daf787-kube-api-access-n62zq\") pod \"f9718008-dd4b-4bb5-b005-395a18daf787\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " Aug 04 19:22:37.988654 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.988571 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9718008-dd4b-4bb5-b005-395a18daf787-extensions-socket-volume\") pod \"f9718008-dd4b-4bb5-b005-395a18daf787\" (UID: \"f9718008-dd4b-4bb5-b005-395a18daf787\") " Aug 04 19:22:37.988974 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.988942 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9718008-dd4b-4bb5-b005-395a18daf787-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f9718008-dd4b-4bb5-b005-395a18daf787" (UID: "f9718008-dd4b-4bb5-b005-395a18daf787"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:22:37.991050 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:37.991016 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9718008-dd4b-4bb5-b005-395a18daf787-kube-api-access-n62zq" (OuterVolumeSpecName: "kube-api-access-n62zq") pod "f9718008-dd4b-4bb5-b005-395a18daf787" (UID: "f9718008-dd4b-4bb5-b005-395a18daf787"). InnerVolumeSpecName "kube-api-access-n62zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:22:38.089711 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.089616 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62zq\" (UniqueName: \"kubernetes.io/projected/f9718008-dd4b-4bb5-b005-395a18daf787-kube-api-access-n62zq\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:22:38.089711 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.089654 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9718008-dd4b-4bb5-b005-395a18daf787-extensions-socket-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:22:38.611624 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.611592 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9718008-dd4b-4bb5-b005-395a18daf787" containerID="6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c" exitCode=0 Aug 04 19:22:38.612058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.611643 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" Aug 04 19:22:38.612058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.611676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" event={"ID":"f9718008-dd4b-4bb5-b005-395a18daf787","Type":"ContainerDied","Data":"6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c"} Aug 04 19:22:38.612058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.611726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h" event={"ID":"f9718008-dd4b-4bb5-b005-395a18daf787","Type":"ContainerDied","Data":"6194dc60ecf22adff0f7361ccaa6f8aa7b95779c286212233285f0d570623cb4"} Aug 04 19:22:38.612058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.611748 2579 scope.go:117] "RemoveContainer" containerID="6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c" Aug 04 19:22:38.623064 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.623040 2579 scope.go:117] "RemoveContainer" containerID="6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c" Aug 04 19:22:38.623391 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:22:38.623366 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c\": container with ID starting with 6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c not found: ID does not exist" containerID="6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c" Aug 04 19:22:38.623495 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.623404 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c"} err="failed to get container status \"6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c\": rpc error: code = NotFound desc = could not find container \"6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c\": container with ID starting with 6a5e4a54e7ed155cd4c4b826a66b8aa5be6a041a39946df5839b94509b8f0f8c not found: ID does not exist" Aug 04 19:22:38.637204 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.637171 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h"] Aug 04 19:22:38.649243 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:38.649210 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-s5b4h"] Aug 04 19:22:39.610449 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:39.610417 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9718008-dd4b-4bb5-b005-395a18daf787" path="/var/lib/kubelet/pods/f9718008-dd4b-4bb5-b005-395a18daf787/volumes" Aug 04 19:22:53.988634 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:53.988552 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m"] Aug 04 19:22:53.989062 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:53.988890 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9718008-dd4b-4bb5-b005-395a18daf787" containerName="manager" Aug 04 19:22:53.989062 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:53.988901 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9718008-dd4b-4bb5-b005-395a18daf787" containerName="manager" Aug 04 19:22:53.989062 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:53.988956 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9718008-dd4b-4bb5-b005-395a18daf787" containerName="manager" Aug 04 19:22:53.990891 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:53.990866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:53.993090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:53.993065 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g5g2q\"" Aug 04 19:22:54.005875 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.005840 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m"] Aug 04 19:22:54.131631 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.131631 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.131881 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.131881 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.131881 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.131881 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.132007 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bx4\" (UniqueName: \"kubernetes.io/projected/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-kube-api-access-59bx4\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.132007 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.131967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.132007 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.132003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233153 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233153 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bx4\" (UniqueName: \"kubernetes.io/projected/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-kube-api-access-59bx4\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233876 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233876 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233876 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.233876 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.233813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.234094 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.234073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.235728 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.235704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.235949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.235929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.243212 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.243150 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.243414 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.243389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bx4\" (UniqueName: \"kubernetes.io/projected/8a771bdd-68af-4d5c-9957-1164e4cfd1c4-kube-api-access-59bx4\") pod \"maas-default-gateway-openshift-default-5ccb968c66-vpq2m\" (UID: \"8a771bdd-68af-4d5c-9957-1164e4cfd1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.301981 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.301948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:54.454206 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.454178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m"] Aug 04 19:22:54.456319 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:22:54.456287 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a771bdd_68af_4d5c_9957_1164e4cfd1c4.slice/crio-3e5d8a64784f6ae8c3e24079be81086c4d3f27c8a0ea04ae198470f2417a1d42 WatchSource:0}: Error finding container 3e5d8a64784f6ae8c3e24079be81086c4d3f27c8a0ea04ae198470f2417a1d42: Status 404 returned error can't find the container with id 3e5d8a64784f6ae8c3e24079be81086c4d3f27c8a0ea04ae198470f2417a1d42 Aug 04 19:22:54.458613 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.458575 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:22:54.458693 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.458640 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:22:54.458693 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.458668 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:22:54.697920 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.697879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" event={"ID":"8a771bdd-68af-4d5c-9957-1164e4cfd1c4","Type":"ContainerStarted","Data":"4c583bd7ba087d6ff360734a3d05b854b889d9b228b898fcfda3012e97e652f5"} Aug 04 19:22:54.698103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.697947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" event={"ID":"8a771bdd-68af-4d5c-9957-1164e4cfd1c4","Type":"ContainerStarted","Data":"3e5d8a64784f6ae8c3e24079be81086c4d3f27c8a0ea04ae198470f2417a1d42"} Aug 04 19:22:54.719562 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:54.719502 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podStartSLOduration=1.719478775 podStartE2EDuration="1.719478775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:22:54.716647964 +0000 UTC m=+451.657444711" watchObservedRunningTime="2026-08-04 19:22:54.719478775 +0000 UTC m=+451.660275520" Aug 04 19:22:55.302420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:55.302379 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:55.307584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:55.307555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:55.702649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:55.702564 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:22:55.703601 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:22:55.703577 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:23:25.474993 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.474953 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-866bbd4fc4-thwhf"] Aug 04 19:23:25.478081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.478051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.480723 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.480700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 19:23:25.481662 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.481644 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dxk86\"" Aug 04 19:23:25.486594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.486569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-866bbd4fc4-thwhf"] Aug 04 19:23:25.606014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.605975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q2b\" (UniqueName: \"kubernetes.io/projected/27f975fb-43f5-4139-8d96-585a778e23c4-kube-api-access-h6q2b\") pod \"authorino-866bbd4fc4-thwhf\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.606175 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.606027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27f975fb-43f5-4139-8d96-585a778e23c4-tls-cert\") pod \"authorino-866bbd4fc4-thwhf\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.707538 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.707500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q2b\" (UniqueName: \"kubernetes.io/projected/27f975fb-43f5-4139-8d96-585a778e23c4-kube-api-access-h6q2b\") pod \"authorino-866bbd4fc4-thwhf\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.707731 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.707588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27f975fb-43f5-4139-8d96-585a778e23c4-tls-cert\") pod \"authorino-866bbd4fc4-thwhf\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.710261 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.710223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27f975fb-43f5-4139-8d96-585a778e23c4-tls-cert\") pod \"authorino-866bbd4fc4-thwhf\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.717276 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.717250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q2b\" (UniqueName: \"kubernetes.io/projected/27f975fb-43f5-4139-8d96-585a778e23c4-kube-api-access-h6q2b\") pod \"authorino-866bbd4fc4-thwhf\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.788468 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.788432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:23:25.929415 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:25.929236 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-866bbd4fc4-thwhf"] Aug 04 19:23:25.932063 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:23:25.932034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f975fb_43f5_4139_8d96_585a778e23c4.slice/crio-8fbfe981708a90f32de370698c10123bc52e8d35084767ee170e4862de15e58b WatchSource:0}: Error finding container 8fbfe981708a90f32de370698c10123bc52e8d35084767ee170e4862de15e58b: Status 404 returned error can't find the container with id 8fbfe981708a90f32de370698c10123bc52e8d35084767ee170e4862de15e58b Aug 04 19:23:26.855544 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:26.855477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" event={"ID":"27f975fb-43f5-4139-8d96-585a778e23c4","Type":"ContainerStarted","Data":"8fbfe981708a90f32de370698c10123bc52e8d35084767ee170e4862de15e58b"} Aug 04 19:23:28.866734 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:28.866684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" event={"ID":"27f975fb-43f5-4139-8d96-585a778e23c4","Type":"ContainerStarted","Data":"26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7"} Aug 04 19:23:28.886244 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:23:28.886190 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" podStartSLOduration=1.844684156 podStartE2EDuration="3.886173852s" totalImagesPullingTime="2.041489696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:23:28.884530484 +0000 UTC m=+485.825327231" watchObservedRunningTime="2026-08-04 19:23:28.886173852 +0000 UTC m=+485.826970597" Aug 04 19:24:17.097984 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:17.097940 2579 generic.go:356] "Generic (PLEG): container finished" podID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerID="4c583bd7ba087d6ff360734a3d05b854b889d9b228b898fcfda3012e97e652f5" exitCode=0 Aug 04 19:24:17.098439 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:17.097984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" event={"ID":"8a771bdd-68af-4d5c-9957-1164e4cfd1c4","Type":"ContainerDied","Data":"4c583bd7ba087d6ff360734a3d05b854b889d9b228b898fcfda3012e97e652f5"} Aug 04 19:24:17.098490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:17.098462 2579 scope.go:117] "RemoveContainer" containerID="4c583bd7ba087d6ff360734a3d05b854b889d9b228b898fcfda3012e97e652f5" Aug 04 19:24:17.099528 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:17.099500 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:24:17.099593 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:17.099572 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:24:17.099627 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:17.099613 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:24:18.105312 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:18.105274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" event={"ID":"8a771bdd-68af-4d5c-9957-1164e4cfd1c4","Type":"ContainerStarted","Data":"e056ddd135f2de6ea8bc14184d1d4cd6900e8c6e756f7ef61564296a7abe0b6f"} Aug 04 19:24:18.302873 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:18.302829 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:24:18.303097 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:18.303077 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vpq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 04 19:24:18.303143 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:18.303124 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podUID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 04 19:24:19.302897 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:19.302853 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vpq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 04 19:24:19.303308 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:19.302914 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podUID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 04 19:24:20.303213 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:20.303172 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vpq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 04 19:24:20.303639 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:20.303234 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podUID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 04 19:24:21.302494 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:21.302453 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vpq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 04 19:24:21.302682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:21.302515 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podUID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 04 19:24:22.302704 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:22.302660 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vpq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 04 19:24:22.303189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:22.302720 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podUID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 04 19:24:23.302698 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:23.302645 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-vpq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 04 19:24:23.303126 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:23.302729 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" podUID="8a771bdd-68af-4d5c-9957-1164e4cfd1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 04 19:24:24.307490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:24.307454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:24:24.307948 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:24.307756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:24:24.308551 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:24.308533 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-vpq2m" Aug 04 19:24:32.187565 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.187528 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n"] Aug 04 19:24:32.191263 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.191235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.200014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.199987 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Aug 04 19:24:32.200843 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.200818 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ppqlf\"" Aug 04 19:24:32.203559 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.203537 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 19:24:32.205054 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.205031 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 19:24:32.214844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.214804 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n"] Aug 04 19:24:32.235209 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.235161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.235209 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.235214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.235465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.235304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.235465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.235366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xkr\" (UniqueName: \"kubernetes.io/projected/9868c8f4-af52-4b34-8c75-97096e41a217-kube-api-access-j8xkr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.235465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.235401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.235465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.235426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9868c8f4-af52-4b34-8c75-97096e41a217-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336376 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336376 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9868c8f4-af52-4b34-8c75-97096e41a217-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xkr\" (UniqueName: \"kubernetes.io/projected/9868c8f4-af52-4b34-8c75-97096e41a217-kube-api-access-j8xkr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336895 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.336976 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.337030 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.336982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.338794 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.338767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9868c8f4-af52-4b34-8c75-97096e41a217-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.339039 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.339021 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9868c8f4-af52-4b34-8c75-97096e41a217-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.349256 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.349215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xkr\" (UniqueName: \"kubernetes.io/projected/9868c8f4-af52-4b34-8c75-97096e41a217-kube-api-access-j8xkr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2n25n\" (UID: \"9868c8f4-af52-4b34-8c75-97096e41a217\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.504231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.504118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:32.660741 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:32.660709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n"] Aug 04 19:24:32.661615 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:24:32.661563 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9868c8f4_af52_4b34_8c75_97096e41a217.slice/crio-367ddd13eae910e56d569ae9eeffa68c75a654eabda9cab7e8749ef161616f3f WatchSource:0}: Error finding container 367ddd13eae910e56d569ae9eeffa68c75a654eabda9cab7e8749ef161616f3f: Status 404 returned error can't find the container with id 367ddd13eae910e56d569ae9eeffa68c75a654eabda9cab7e8749ef161616f3f Aug 04 19:24:33.183783 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:33.183737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" event={"ID":"9868c8f4-af52-4b34-8c75-97096e41a217","Type":"ContainerStarted","Data":"367ddd13eae910e56d569ae9eeffa68c75a654eabda9cab7e8749ef161616f3f"} Aug 04 19:24:34.189760 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:34.189660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" event={"ID":"9868c8f4-af52-4b34-8c75-97096e41a217","Type":"ContainerStarted","Data":"7298081b336fa56facf564edec3b025309dccfc3b6005b2f452aca52e0f6a7af"} Aug 04 19:24:34.209589 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:34.209531 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" podStartSLOduration=1.005046549 podStartE2EDuration="2.209511661s" totalImagesPullingTime="1.204465112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:24:34.208171015 +0000 UTC m=+551.148967765" watchObservedRunningTime="2026-08-04 19:24:34.209511661 +0000 UTC m=+551.150308406" Aug 04 19:24:42.504961 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:42.504904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:42.505479 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:42.504998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:42.517470 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:42.517443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:24:43.246072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:24:43.246041 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2n25n" Aug 04 19:25:23.524922 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:23.524883 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:25:23.528195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:23.528166 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:25:24.690230 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.690186 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n"] Aug 04 19:25:24.692729 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.692701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.695481 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.695453 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 19:25:24.706126 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.706092 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n"] Aug 04 19:25:24.709230 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.709197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.709414 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.709244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.709414 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.709368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.709521 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.709466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.709573 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.709544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b10b9f-6c58-409b-92bd-5272f14d5e57-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.709625 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.709605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q5z\" (UniqueName: \"kubernetes.io/projected/e0b10b9f-6c58-409b-92bd-5272f14d5e57-kube-api-access-h6q5z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.810728 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.810686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b10b9f-6c58-409b-92bd-5272f14d5e57-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.810728 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.810730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q5z\" (UniqueName: \"kubernetes.io/projected/e0b10b9f-6c58-409b-92bd-5272f14d5e57-kube-api-access-h6q5z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.810985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.810773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.810985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.810794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.810985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.810823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.810985 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.810848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.811253 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.811230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.811321 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.811272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.811321 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.811292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.813183 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.813144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b10b9f-6c58-409b-92bd-5272f14d5e57-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.813420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.813400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b10b9f-6c58-409b-92bd-5272f14d5e57-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:24.819373 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:24.819341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q5z\" (UniqueName: \"kubernetes.io/projected/e0b10b9f-6c58-409b-92bd-5272f14d5e57-kube-api-access-h6q5z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n\" (UID: \"e0b10b9f-6c58-409b-92bd-5272f14d5e57\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:25.003582 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:25.003475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:25.156569 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:25.156534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n"] Aug 04 19:25:25.158799 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:25:25.158764 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b10b9f_6c58_409b_92bd_5272f14d5e57.slice/crio-2849b9237b751388887d8940fd8874441cd53fa1f48f2cd812c1dac242018f37 WatchSource:0}: Error finding container 2849b9237b751388887d8940fd8874441cd53fa1f48f2cd812c1dac242018f37: Status 404 returned error can't find the container with id 2849b9237b751388887d8940fd8874441cd53fa1f48f2cd812c1dac242018f37 Aug 04 19:25:25.441012 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:25.440973 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" event={"ID":"e0b10b9f-6c58-409b-92bd-5272f14d5e57","Type":"ContainerStarted","Data":"2849b9237b751388887d8940fd8874441cd53fa1f48f2cd812c1dac242018f37"} Aug 04 19:25:26.448767 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:26.448722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" event={"ID":"e0b10b9f-6c58-409b-92bd-5272f14d5e57","Type":"ContainerStarted","Data":"a2c15ad65e8d49807d87e79fbfa848b57472f78405b3d26409a7b47b9223df62"} Aug 04 19:25:26.469287 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:26.469229 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" podStartSLOduration=2.238368944 podStartE2EDuration="2.469209855s" totalImagesPullingTime="230.840911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:25:26.467685304 +0000 UTC m=+603.408482051" watchObservedRunningTime="2026-08-04 19:25:26.469209855 +0000 UTC m=+603.410006602" Aug 04 19:25:32.171497 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.171454 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm"] Aug 04 19:25:32.174320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.174294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.176646 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.176621 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 04 19:25:32.185296 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.185265 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm"] Aug 04 19:25:32.277806 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.277767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.278015 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.277818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddllr\" (UniqueName: \"kubernetes.io/projected/3c16efa8-01a3-4722-9703-fc93b9de8cb6-kube-api-access-ddllr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.278015 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.277900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c16efa8-01a3-4722-9703-fc93b9de8cb6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.278015 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.277966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.278015 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.278015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.278219 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.278036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.378800 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.378761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.378986 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.378806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.378986 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.378864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.378986 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.378901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddllr\" (UniqueName: \"kubernetes.io/projected/3c16efa8-01a3-4722-9703-fc93b9de8cb6-kube-api-access-ddllr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.378986 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.378950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c16efa8-01a3-4722-9703-fc93b9de8cb6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.379195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.379007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.379289 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.379268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.379418 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.379397 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.379477 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.379435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.381358 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.381315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c16efa8-01a3-4722-9703-fc93b9de8cb6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.381609 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.381587 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c16efa8-01a3-4722-9703-fc93b9de8cb6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.387576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.387548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddllr\" (UniqueName: \"kubernetes.io/projected/3c16efa8-01a3-4722-9703-fc93b9de8cb6-kube-api-access-ddllr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-czsqm\" (UID: \"3c16efa8-01a3-4722-9703-fc93b9de8cb6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.486685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.486579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:32.635770 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.635740 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm"] Aug 04 19:25:32.639018 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:25:32.638982 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c16efa8_01a3_4722_9703_fc93b9de8cb6.slice/crio-32d545b0bc4dc1380fe36d86573003a4a12544e97946233670387ef49af01b62 WatchSource:0}: Error finding container 32d545b0bc4dc1380fe36d86573003a4a12544e97946233670387ef49af01b62: Status 404 returned error can't find the container with id 32d545b0bc4dc1380fe36d86573003a4a12544e97946233670387ef49af01b62 Aug 04 19:25:32.640509 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:32.640484 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 19:25:33.483671 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:33.483632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" event={"ID":"3c16efa8-01a3-4722-9703-fc93b9de8cb6","Type":"ContainerStarted","Data":"7067ca1ac990d964bd4402eb60dc2dc25fe45fffaa50b7f72f9e86dd8b821e18"} Aug 04 19:25:33.484059 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:33.483692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" event={"ID":"3c16efa8-01a3-4722-9703-fc93b9de8cb6","Type":"ContainerStarted","Data":"32d545b0bc4dc1380fe36d86573003a4a12544e97946233670387ef49af01b62"} Aug 04 19:25:33.503637 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:33.503583 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" podStartSLOduration=1.299096018 podStartE2EDuration="1.503563782s" totalImagesPullingTime="204.467764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:25:33.501933077 +0000 UTC m=+610.442729835" watchObservedRunningTime="2026-08-04 19:25:33.503563782 +0000 UTC m=+610.444360527" Aug 04 19:25:35.004080 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:35.004014 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:35.004642 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:35.004123 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:35.024211 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:35.024173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:35.506494 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:35.506464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n" Aug 04 19:25:42.487472 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:42.487427 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:42.487938 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:42.487496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:42.500317 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:42.500279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:25:42.537369 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:25:42.537310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-czsqm" Aug 04 19:26:42.519694 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.519657 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866bbd4fc4-thwhf"] Aug 04 19:26:42.520282 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.519889 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" podUID="27f975fb-43f5-4139-8d96-585a778e23c4" containerName="authorino" containerID="cri-o://26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7" gracePeriod=30 Aug 04 19:26:42.790465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.790437 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:26:42.819306 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.819266 2579 generic.go:356] "Generic (PLEG): container finished" podID="27f975fb-43f5-4139-8d96-585a778e23c4" containerID="26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7" exitCode=0 Aug 04 19:26:42.819520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.819363 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" Aug 04 19:26:42.819520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.819370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" event={"ID":"27f975fb-43f5-4139-8d96-585a778e23c4","Type":"ContainerDied","Data":"26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7"} Aug 04 19:26:42.819520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.819417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-866bbd4fc4-thwhf" event={"ID":"27f975fb-43f5-4139-8d96-585a778e23c4","Type":"ContainerDied","Data":"8fbfe981708a90f32de370698c10123bc52e8d35084767ee170e4862de15e58b"} Aug 04 19:26:42.819520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.819440 2579 scope.go:117] "RemoveContainer" containerID="26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7" Aug 04 19:26:42.830094 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.830066 2579 scope.go:117] "RemoveContainer" containerID="26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7" Aug 04 19:26:42.830469 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:26:42.830447 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7\": container with ID starting with 26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7 not found: ID does not exist" containerID="26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7" Aug 04 19:26:42.830538 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.830479 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7"} err="failed to get container status \"26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7\": rpc error: code = NotFound desc = could not find container \"26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7\": container with ID starting with 26e4b5ba7c470a613e6bacd0ef9b3a9670e53c923856f1bc2d0b0577957847e7 not found: ID does not exist" Aug 04 19:26:42.918768 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.918713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27f975fb-43f5-4139-8d96-585a778e23c4-tls-cert\") pod \"27f975fb-43f5-4139-8d96-585a778e23c4\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " Aug 04 19:26:42.918971 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.918837 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6q2b\" (UniqueName: \"kubernetes.io/projected/27f975fb-43f5-4139-8d96-585a778e23c4-kube-api-access-h6q2b\") pod \"27f975fb-43f5-4139-8d96-585a778e23c4\" (UID: \"27f975fb-43f5-4139-8d96-585a778e23c4\") " Aug 04 19:26:42.921117 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.921084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f975fb-43f5-4139-8d96-585a778e23c4-kube-api-access-h6q2b" (OuterVolumeSpecName: "kube-api-access-h6q2b") pod "27f975fb-43f5-4139-8d96-585a778e23c4" (UID: "27f975fb-43f5-4139-8d96-585a778e23c4"). InnerVolumeSpecName "kube-api-access-h6q2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:26:42.931013 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:42.930969 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f975fb-43f5-4139-8d96-585a778e23c4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "27f975fb-43f5-4139-8d96-585a778e23c4" (UID: "27f975fb-43f5-4139-8d96-585a778e23c4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:26:43.019760 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:43.019721 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6q2b\" (UniqueName: \"kubernetes.io/projected/27f975fb-43f5-4139-8d96-585a778e23c4-kube-api-access-h6q2b\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:26:43.019760 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:43.019751 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/27f975fb-43f5-4139-8d96-585a778e23c4-tls-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:26:43.141593 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:43.141556 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-866bbd4fc4-thwhf"] Aug 04 19:26:43.149400 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:43.149362 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-866bbd4fc4-thwhf"] Aug 04 19:26:43.615529 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:26:43.612078 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f975fb-43f5-4139-8d96-585a778e23c4" path="/var/lib/kubelet/pods/27f975fb-43f5-4139-8d96-585a778e23c4/volumes" Aug 04 19:30:23.556648 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:30:23.556567 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:30:23.557164 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:30:23.556983 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:35:23.585113 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:23.585079 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:35:23.587660 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:23.586149 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:35:25.068403 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.068359 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz"] Aug 04 19:35:25.068880 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.068583 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" podUID="1de813bd-c7f2-46d3-a22c-31e79fa9ce97" containerName="manager" containerID="cri-o://acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b" gracePeriod=10 Aug 04 19:35:25.327824 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.327744 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:35:25.375390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.375323 2579 generic.go:356] "Generic (PLEG): container finished" podID="1de813bd-c7f2-46d3-a22c-31e79fa9ce97" containerID="acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b" exitCode=0 Aug 04 19:35:25.375587 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.375404 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" Aug 04 19:35:25.375587 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.375418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" event={"ID":"1de813bd-c7f2-46d3-a22c-31e79fa9ce97","Type":"ContainerDied","Data":"acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b"} Aug 04 19:35:25.375587 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.375480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz" event={"ID":"1de813bd-c7f2-46d3-a22c-31e79fa9ce97","Type":"ContainerDied","Data":"fa051e794efc9a8a1404ac2e1f2d0e1a44504fb4dbd23d41a3d28625af5c4f9a"} Aug 04 19:35:25.375587 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.375503 2579 scope.go:117] "RemoveContainer" containerID="acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b" Aug 04 19:35:25.387635 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.387609 2579 scope.go:117] "RemoveContainer" containerID="acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b" Aug 04 19:35:25.387997 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:35:25.387975 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b\": container with ID starting with acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b not found: ID does not exist" containerID="acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b" Aug 04 19:35:25.388057 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.388008 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b"} err="failed to get container status \"acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b\": rpc error: code = NotFound desc = could not find container \"acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b\": container with ID starting with acc201c82470dcd99d24b18f071432485d5613a1b74ab4a6f65a232505092d2b not found: ID does not exist" Aug 04 19:35:25.461024 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.460990 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztppd\" (UniqueName: \"kubernetes.io/projected/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-kube-api-access-ztppd\") pod \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " Aug 04 19:35:25.461231 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.461074 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-extensions-socket-volume\") pod \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\" (UID: \"1de813bd-c7f2-46d3-a22c-31e79fa9ce97\") " Aug 04 19:35:25.461526 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.461493 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1de813bd-c7f2-46d3-a22c-31e79fa9ce97" (UID: "1de813bd-c7f2-46d3-a22c-31e79fa9ce97"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:35:25.463191 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.463158 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-kube-api-access-ztppd" (OuterVolumeSpecName: "kube-api-access-ztppd") pod "1de813bd-c7f2-46d3-a22c-31e79fa9ce97" (UID: "1de813bd-c7f2-46d3-a22c-31e79fa9ce97"). InnerVolumeSpecName "kube-api-access-ztppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:35:25.562001 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.561960 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztppd\" (UniqueName: \"kubernetes.io/projected/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-kube-api-access-ztppd\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:35:25.562001 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.561992 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1de813bd-c7f2-46d3-a22c-31e79fa9ce97-extensions-socket-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:35:25.692836 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.692792 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz"] Aug 04 19:35:25.698428 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:25.698387 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6nqdz"] Aug 04 19:35:27.615296 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:35:27.611466 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de813bd-c7f2-46d3-a22c-31e79fa9ce97" path="/var/lib/kubelet/pods/1de813bd-c7f2-46d3-a22c-31e79fa9ce97/volumes" Aug 04 19:36:25.746048 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746012 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj"] Aug 04 19:36:25.746534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746389 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de813bd-c7f2-46d3-a22c-31e79fa9ce97" containerName="manager" Aug 04 19:36:25.746534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746406 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de813bd-c7f2-46d3-a22c-31e79fa9ce97" containerName="manager" Aug 04 19:36:25.746534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746430 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f975fb-43f5-4139-8d96-585a778e23c4" containerName="authorino" Aug 04 19:36:25.746534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746435 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f975fb-43f5-4139-8d96-585a778e23c4" containerName="authorino" Aug 04 19:36:25.746534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746503 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de813bd-c7f2-46d3-a22c-31e79fa9ce97" containerName="manager" Aug 04 19:36:25.746534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.746512 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f975fb-43f5-4139-8d96-585a778e23c4" containerName="authorino" Aug 04 19:36:25.749865 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.749845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:25.752739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.752713 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-846sv\"" Aug 04 19:36:25.767510 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.767476 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj"] Aug 04 19:36:25.841681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.841642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4ftnj\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:25.841880 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.841702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h6p\" (UniqueName: \"kubernetes.io/projected/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-kube-api-access-28h6p\") pod \"kuadrant-operator-controller-manager-68cf589589-4ftnj\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:25.943371 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.943308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4ftnj\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:25.943535 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.943388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h6p\" (UniqueName: \"kubernetes.io/projected/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-kube-api-access-28h6p\") pod \"kuadrant-operator-controller-manager-68cf589589-4ftnj\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:25.943712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.943692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4ftnj\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:25.951478 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:25.951449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h6p\" (UniqueName: \"kubernetes.io/projected/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-kube-api-access-28h6p\") pod \"kuadrant-operator-controller-manager-68cf589589-4ftnj\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:26.060416 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:26.060373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:26.207525 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:26.207484 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj"] Aug 04 19:36:26.210893 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:36:26.210865 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b78b673_4f3d_4caa_8c2b_f7025bd2a930.slice/crio-712a63107513aad2053168ce4992a83e8359fb7c59f6021337c3b1fa6af47e60 WatchSource:0}: Error finding container 712a63107513aad2053168ce4992a83e8359fb7c59f6021337c3b1fa6af47e60: Status 404 returned error can't find the container with id 712a63107513aad2053168ce4992a83e8359fb7c59f6021337c3b1fa6af47e60 Aug 04 19:36:26.675565 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:26.675520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" event={"ID":"8b78b673-4f3d-4caa-8c2b-f7025bd2a930","Type":"ContainerStarted","Data":"c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140"} Aug 04 19:36:26.675768 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:26.675601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:36:26.675768 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:26.675628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" event={"ID":"8b78b673-4f3d-4caa-8c2b-f7025bd2a930","Type":"ContainerStarted","Data":"712a63107513aad2053168ce4992a83e8359fb7c59f6021337c3b1fa6af47e60"} Aug 04 19:36:26.698539 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:26.698485 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" podStartSLOduration=1.698466009 podStartE2EDuration="1.698466009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:36:26.696406708 +0000 UTC m=+1263.637203451" watchObservedRunningTime="2026-08-04 19:36:26.698466009 +0000 UTC m=+1263.639262755" Aug 04 19:36:37.683347 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:36:37.683305 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:39:42.997773 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:42.997737 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp"] Aug 04 19:39:43.001162 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.001142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.003521 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.003495 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3fe4ffd9-kserve-self-signed-certs\"" Aug 04 19:39:43.019754 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.019715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp"] Aug 04 19:39:43.115323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.115275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tmp-dir\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.115323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.115319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tls-certs\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.115585 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.115391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-home\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.115585 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.115470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4phb\" (UniqueName: \"kubernetes.io/projected/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-kube-api-access-g4phb\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.115585 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.115505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-dshm\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.115585 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.115524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-model-cache\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4phb\" (UniqueName: \"kubernetes.io/projected/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-kube-api-access-g4phb\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216267 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-dshm\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216267 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-model-cache\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216267 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tmp-dir\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216267 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tls-certs\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216267 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-home\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216659 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tmp-dir\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216719 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-home\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.216754 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.216712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-model-cache\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.218551 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.218521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-dshm\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.218756 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.218739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tls-certs\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.223878 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.223849 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4phb\" (UniqueName: \"kubernetes.io/projected/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-kube-api-access-g4phb\") pod \"e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.310651 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.310613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:43.459907 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.459873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp"] Aug 04 19:39:43.462553 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:39:43.462520 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0795ac2f_fd2b_41e5_9c12_f020c88bf6db.slice/crio-0678a6ac6e5c4e10903e7fad374a172a703ba2d83b639de7d46835db8aaa1c84 WatchSource:0}: Error finding container 0678a6ac6e5c4e10903e7fad374a172a703ba2d83b639de7d46835db8aaa1c84: Status 404 returned error can't find the container with id 0678a6ac6e5c4e10903e7fad374a172a703ba2d83b639de7d46835db8aaa1c84 Aug 04 19:39:43.668856 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.668742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" event={"ID":"0795ac2f-fd2b-41e5-9c12-f020c88bf6db","Type":"ContainerStarted","Data":"38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0"} Aug 04 19:39:43.668856 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.668799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" event={"ID":"0795ac2f-fd2b-41e5-9c12-f020c88bf6db","Type":"ContainerStarted","Data":"0678a6ac6e5c4e10903e7fad374a172a703ba2d83b639de7d46835db8aaa1c84"} Aug 04 19:39:43.686905 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:43.686846 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" podStartSLOduration=1.686827641 podStartE2EDuration="1.686827641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:39:43.685561243 +0000 UTC m=+1460.626357989" watchObservedRunningTime="2026-08-04 19:39:43.686827641 +0000 UTC m=+1460.627624386" Aug 04 19:39:53.311502 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:53.311459 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:53.311968 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:53.311523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:53.324186 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:53.324156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:39:53.731264 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:39:53.731173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:40:06.467153 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:06.467119 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp"] Aug 04 19:40:21.488175 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.488129 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" podUID="0795ac2f-fd2b-41e5-9c12-f020c88bf6db" containerName="main" containerID="cri-o://38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0" gracePeriod=45 Aug 04 19:40:21.751805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.751735 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:40:21.783996 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.783959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tmp-dir\") pod \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " Aug 04 19:40:21.783996 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784002 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-home\") pod \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " Aug 04 19:40:21.784275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784030 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-dshm\") pod \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " Aug 04 19:40:21.784275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-model-cache\") pod \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " Aug 04 19:40:21.784275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4phb\" (UniqueName: \"kubernetes.io/projected/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-kube-api-access-g4phb\") pod \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " Aug 04 19:40:21.784275 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tls-certs\") pod \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\" (UID: \"0795ac2f-fd2b-41e5-9c12-f020c88bf6db\") " Aug 04 19:40:21.784520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0795ac2f-fd2b-41e5-9c12-f020c88bf6db" (UID: "0795ac2f-fd2b-41e5-9c12-f020c88bf6db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:21.784520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784405 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-home" (OuterVolumeSpecName: "home") pod "0795ac2f-fd2b-41e5-9c12-f020c88bf6db" (UID: "0795ac2f-fd2b-41e5-9c12-f020c88bf6db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:21.784520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784446 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-model-cache" (OuterVolumeSpecName: "model-cache") pod "0795ac2f-fd2b-41e5-9c12-f020c88bf6db" (UID: "0795ac2f-fd2b-41e5-9c12-f020c88bf6db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:21.784520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784480 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:40:21.784520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.784501 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:40:21.786638 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.786614 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-dshm" (OuterVolumeSpecName: "dshm") pod "0795ac2f-fd2b-41e5-9c12-f020c88bf6db" (UID: "0795ac2f-fd2b-41e5-9c12-f020c88bf6db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:40:21.786781 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.786706 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0795ac2f-fd2b-41e5-9c12-f020c88bf6db" (UID: "0795ac2f-fd2b-41e5-9c12-f020c88bf6db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:40:21.786837 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.786791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-kube-api-access-g4phb" (OuterVolumeSpecName: "kube-api-access-g4phb") pod "0795ac2f-fd2b-41e5-9c12-f020c88bf6db" (UID: "0795ac2f-fd2b-41e5-9c12-f020c88bf6db"). InnerVolumeSpecName "kube-api-access-g4phb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:40:21.863545 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.863506 2579 generic.go:356] "Generic (PLEG): container finished" podID="0795ac2f-fd2b-41e5-9c12-f020c88bf6db" containerID="38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0" exitCode=0 Aug 04 19:40:21.863545 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.863549 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" Aug 04 19:40:21.863743 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.863597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" event={"ID":"0795ac2f-fd2b-41e5-9c12-f020c88bf6db","Type":"ContainerDied","Data":"38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0"} Aug 04 19:40:21.863743 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.863639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp" event={"ID":"0795ac2f-fd2b-41e5-9c12-f020c88bf6db","Type":"ContainerDied","Data":"0678a6ac6e5c4e10903e7fad374a172a703ba2d83b639de7d46835db8aaa1c84"} Aug 04 19:40:21.863743 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.863658 2579 scope.go:117] "RemoveContainer" containerID="38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0" Aug 04 19:40:21.873138 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.873114 2579 scope.go:117] "RemoveContainer" containerID="38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0" Aug 04 19:40:21.873483 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:40:21.873463 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0\": container with ID starting with 38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0 not found: ID does not exist" containerID="38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0" Aug 04 19:40:21.873539 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.873494 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0"} err="failed to get container status \"38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0\": rpc error: code = NotFound desc = could not find container \"38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0\": container with ID starting with 38d0705360012aa4eb3ecba47547b6e8ff5702ea7e31301e890a198a0441d2d0 not found: ID does not exist" Aug 04 19:40:21.884317 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.884280 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp"] Aug 04 19:40:21.885226 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.885205 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:40:21.885272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.885229 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:40:21.885272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.885239 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4phb\" (UniqueName: \"kubernetes.io/projected/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-kube-api-access-g4phb\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:40:21.885272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.885248 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795ac2f-fd2b-41e5-9c12-f020c88bf6db-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:40:21.889703 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:21.889675 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3fe4ffd9-kserve-5c9d794f47-x9sjp"] Aug 04 19:40:23.612011 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:23.611897 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0795ac2f-fd2b-41e5-9c12-f020c88bf6db" path="/var/lib/kubelet/pods/0795ac2f-fd2b-41e5-9c12-f020c88bf6db/volumes" Aug 04 19:40:23.621268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:23.618231 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:40:23.621268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:40:23.619722 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:44:01.411485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.411405 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4"] Aug 04 19:44:01.411910 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.411828 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0795ac2f-fd2b-41e5-9c12-f020c88bf6db" containerName="main" Aug 04 19:44:01.411910 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.411842 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0795ac2f-fd2b-41e5-9c12-f020c88bf6db" containerName="main" Aug 04 19:44:01.411983 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.411914 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0795ac2f-fd2b-41e5-9c12-f020c88bf6db" containerName="main" Aug 04 19:44:01.413830 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.413814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.416251 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.416213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89a8e771-openshift-default-dockercfg-l89ch\"" Aug 04 19:44:01.434968 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.434930 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4"] Aug 04 19:44:01.509594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.509549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-credential-socket\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.509811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.509613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-podinfo\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.509811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.509762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-data\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.509941 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.509841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-envoy\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.509941 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.509903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac89d169-a83f-432e-bd4e-cd0bf0141331-istiod-ca-cert\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.510054 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.509965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-token\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.510054 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.510036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-socket\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.510160 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.510116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27w6\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-kube-api-access-w27w6\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.510219 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.510174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-certs\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614482 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27w6\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-kube-api-access-w27w6\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614482 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-certs\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-credential-socket\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-podinfo\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-data\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-envoy\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614934 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac89d169-a83f-432e-bd4e-cd0bf0141331-istiod-ca-cert\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614934 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-token\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.614934 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.614814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-socket\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.615085 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.615034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-certs\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.615141 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.615121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-credential-socket\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.615141 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.615131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-socket\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.615373 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.615323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-data\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.615751 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.615729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac89d169-a83f-432e-bd4e-cd0bf0141331-istiod-ca-cert\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.617248 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.617226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-envoy\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.617649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.617630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-podinfo\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.625579 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.625529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-token\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.625716 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.625580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27w6\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-kube-api-access-w27w6\") pod \"e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.724455 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.724369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:01.871854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.871710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4"] Aug 04 19:44:01.874968 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:44:01.874929 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac89d169_a83f_432e_bd4e_cd0bf0141331.slice/crio-f0bde2ffd061aa99ca3fae7248e095c3b26d37def7543277e2881d4d4cfaa50c WatchSource:0}: Error finding container f0bde2ffd061aa99ca3fae7248e095c3b26d37def7543277e2881d4d4cfaa50c: Status 404 returned error can't find the container with id f0bde2ffd061aa99ca3fae7248e095c3b26d37def7543277e2881d4d4cfaa50c Aug 04 19:44:01.877167 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.877136 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:01.877258 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.877222 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:01.877323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.877267 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:01.958928 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.958879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" event={"ID":"ac89d169-a83f-432e-bd4e-cd0bf0141331","Type":"ContainerStarted","Data":"6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e"} Aug 04 19:44:01.959116 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.958954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" event={"ID":"ac89d169-a83f-432e-bd4e-cd0bf0141331","Type":"ContainerStarted","Data":"f0bde2ffd061aa99ca3fae7248e095c3b26d37def7543277e2881d4d4cfaa50c"} Aug 04 19:44:01.983034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:01.982922 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podStartSLOduration=0.982901235 podStartE2EDuration="982.901235ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:01.979781883 +0000 UTC m=+1718.920578630" watchObservedRunningTime="2026-08-04 19:44:01.982901235 +0000 UTC m=+1718.923697980" Aug 04 19:44:02.725515 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:02.725469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:02.730485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:02.730457 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:02.964285 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:02.964242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:02.965262 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:02.965240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:17.966111 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:17.966070 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 04 19:44:17.966542 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:17.966145 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 04 19:44:18.040476 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:18.040443 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerID="6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e" exitCode=0 Aug 04 19:44:18.040664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:18.040522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" event={"ID":"ac89d169-a83f-432e-bd4e-cd0bf0141331","Type":"ContainerDied","Data":"6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e"} Aug 04 19:44:18.040952 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:18.040937 2579 scope.go:117] "RemoveContainer" containerID="6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e" Aug 04 19:44:18.041993 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:18.041968 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:18.042053 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:18.042034 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:18.042096 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:18.042066 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:19.051415 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:19.051369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" event={"ID":"ac89d169-a83f-432e-bd4e-cd0bf0141331","Type":"ContainerStarted","Data":"528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6"} Aug 04 19:44:19.725159 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:19.725121 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:19.725364 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:19.725300 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 04 19:44:19.725407 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:19.725371 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 04 19:44:20.725129 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:20.725089 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 04 19:44:20.725568 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:20.725153 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 04 19:44:21.725113 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:21.725072 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 04 19:44:21.725268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:21.725133 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 04 19:44:22.725622 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:22.725576 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 04 19:44:22.726001 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:22.725640 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 04 19:44:23.725269 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:23.725227 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 04 19:44:23.725483 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:23.725285 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 04 19:44:24.729485 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:24.729454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:24.729877 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:24.729744 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:24.730470 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:24.730453 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:35.368304 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:35.368264 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4"] Aug 04 19:44:35.368864 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:35.368606 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" containerID="cri-o://528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6" gracePeriod=30 Aug 04 19:44:36.351141 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.351101 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc"] Aug 04 19:44:36.353830 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.353803 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.357453 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:44:36.357413 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" is forbidden: User \"system:node:ip-10-0-143-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-62.ec2.internal' and this object" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.357848 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.357827 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ead82904-gw-openshift-default-dockercfg-p5f6d\"" Aug 04 19:44:36.377419 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.377380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc"] Aug 04 19:44:36.470320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-podinfo\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470531 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-credential-socket\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470531 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-certs\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470531 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-socket\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-envoy\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6rb\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-kube-api-access-ld6rb\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-token\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470808 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-data\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.470808 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.470751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570af46d-2f6b-4eab-83ad-d80d90696ab8-istiod-ca-cert\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.571586 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-token\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.571791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-data\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.571791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570af46d-2f6b-4eab-83ad-d80d90696ab8-istiod-ca-cert\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.571791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-podinfo\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.571791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-credential-socket\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.571791 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-certs\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572077 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-socket\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572077 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-envoy\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572077 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.571895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6rb\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-kube-api-access-ld6rb\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572254 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.572100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-data\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572254 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.572199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-socket\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572392 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.572302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-certs\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572445 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.572407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-credential-socket\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.572591 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.572572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570af46d-2f6b-4eab-83ad-d80d90696ab8-istiod-ca-cert\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.574187 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.574167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-envoy\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.574473 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.574456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-podinfo\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.580704 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.580666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-token\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.580704 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.580677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6rb\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-kube-api-access-ld6rb\") pod \"e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.664790 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.664673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:36.806631 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.806525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc"] Aug 04 19:44:36.809633 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:44:36.809602 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570af46d_2f6b_4eab_83ad_d80d90696ab8.slice/crio-3e2ea4a61f3dd044f83d636aaa22838c50f4f9d517a4a45bf6078c5642b227d1 WatchSource:0}: Error finding container 3e2ea4a61f3dd044f83d636aaa22838c50f4f9d517a4a45bf6078c5642b227d1: Status 404 returned error can't find the container with id 3e2ea4a61f3dd044f83d636aaa22838c50f4f9d517a4a45bf6078c5642b227d1 Aug 04 19:44:36.811727 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.811697 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:36.811811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.811772 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:36.811846 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:36.811807 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:37.144310 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:37.144266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" event={"ID":"570af46d-2f6b-4eab-83ad-d80d90696ab8","Type":"ContainerStarted","Data":"c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a"} Aug 04 19:44:37.144623 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:37.144392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" event={"ID":"570af46d-2f6b-4eab-83ad-d80d90696ab8","Type":"ContainerStarted","Data":"3e2ea4a61f3dd044f83d636aaa22838c50f4f9d517a4a45bf6078c5642b227d1"} Aug 04 19:44:37.182035 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:37.181971 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podStartSLOduration=1.181953183 podStartE2EDuration="1.181953183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:44:37.179387669 +0000 UTC m=+1754.120184416" watchObservedRunningTime="2026-08-04 19:44:37.181953183 +0000 UTC m=+1754.122749929" Aug 04 19:44:37.665306 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:37.665264 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:37.670457 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:37.670428 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:38.149949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:38.149917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:38.150932 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:38.150899 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:39.732957 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:39.732912 2579 patch_prober.go:28] interesting pod/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:44:39.733358 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:39.732967 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:44:40.636440 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.636413 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:40.811132 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811091 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-data\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac89d169-a83f-432e-bd4e-cd0bf0141331-istiod-ca-cert\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811168 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-token\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811213 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-envoy\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811293 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-socket\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w27w6\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-kube-api-access-w27w6\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-certs\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811439 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811454 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-credential-socket\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.811649 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811490 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-podinfo\") pod \"ac89d169-a83f-432e-bd4e-cd0bf0141331\" (UID: \"ac89d169-a83f-432e-bd4e-cd0bf0141331\") " Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811680 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811700 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac89d169-a83f-432e-bd4e-cd0bf0141331-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811892 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811912 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac89d169-a83f-432e-bd4e-cd0bf0141331-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811920 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811927 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.812069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.811957 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.813684 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.813652 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:44:40.813684 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.813675 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:44:40.813879 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.813858 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:44:40.813926 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.813863 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-kube-api-access-w27w6" (OuterVolumeSpecName: "kube-api-access-w27w6") pod "ac89d169-a83f-432e-bd4e-cd0bf0141331" (UID: "ac89d169-a83f-432e-bd4e-cd0bf0141331"). InnerVolumeSpecName "kube-api-access-w27w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:44:40.913252 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.913215 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.913252 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.913248 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27w6\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-kube-api-access-w27w6\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.913252 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.913262 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac89d169-a83f-432e-bd4e-cd0bf0141331-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.913513 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.913274 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:40.913513 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:40.913285 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac89d169-a83f-432e-bd4e-cd0bf0141331-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:44:41.166426 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.166317 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerID="528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6" exitCode=0 Aug 04 19:44:41.166426 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.166398 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" Aug 04 19:44:41.166426 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.166418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" event={"ID":"ac89d169-a83f-432e-bd4e-cd0bf0141331","Type":"ContainerDied","Data":"528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6"} Aug 04 19:44:41.166677 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.166498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4" event={"ID":"ac89d169-a83f-432e-bd4e-cd0bf0141331","Type":"ContainerDied","Data":"f0bde2ffd061aa99ca3fae7248e095c3b26d37def7543277e2881d4d4cfaa50c"} Aug 04 19:44:41.166677 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.166522 2579 scope.go:117] "RemoveContainer" containerID="528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6" Aug 04 19:44:41.176828 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.176810 2579 scope.go:117] "RemoveContainer" containerID="6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e" Aug 04 19:44:41.185264 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.185244 2579 scope.go:117] "RemoveContainer" containerID="528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6" Aug 04 19:44:41.185566 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:44:41.185544 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6\": container with ID starting with 528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6 not found: ID does not exist" containerID="528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6" Aug 04 19:44:41.185617 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.185576 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6"} err="failed to get container status \"528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6\": rpc error: code = NotFound desc = could not find container \"528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6\": container with ID starting with 528a1648690c415bbe1a52938151d44bfed4cf10598e873fd61d0072ee4c67c6 not found: ID does not exist" Aug 04 19:44:41.185617 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.185597 2579 scope.go:117] "RemoveContainer" containerID="6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e" Aug 04 19:44:41.185845 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:44:41.185824 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e\": container with ID starting with 6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e not found: ID does not exist" containerID="6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e" Aug 04 19:44:41.185887 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.185851 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e"} err="failed to get container status \"6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e\": rpc error: code = NotFound desc = could not find container \"6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e\": container with ID starting with 6aac98ad25643514bacc5185ee6c59fdf7aab62880a6dde4a47d8a87857c679e not found: ID does not exist" Aug 04 19:44:41.189410 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.189383 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4"] Aug 04 19:44:41.192817 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.192791 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89a8e771-openshift-default-54d594fc48-w8sp4"] Aug 04 19:44:41.610018 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:41.609981 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" path="/var/lib/kubelet/pods/ac89d169-a83f-432e-bd4e-cd0bf0141331/volumes" Aug 04 19:44:52.223663 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:52.223577 2579 generic.go:356] "Generic (PLEG): container finished" podID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerID="c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a" exitCode=0 Aug 04 19:44:52.223663 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:52.223647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" event={"ID":"570af46d-2f6b-4eab-83ad-d80d90696ab8","Type":"ContainerDied","Data":"c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a"} Aug 04 19:44:52.224158 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:52.224059 2579 scope.go:117] "RemoveContainer" containerID="c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a" Aug 04 19:44:52.225127 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:52.225097 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:52.225220 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:52.225163 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:52.225220 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:52.225197 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:44:53.150394 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:53.150310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:53.230884 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:53.230842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" event={"ID":"570af46d-2f6b-4eab-83ad-d80d90696ab8","Type":"ContainerStarted","Data":"edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7"} Aug 04 19:44:53.665072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:53.665030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:53.666459 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:53.666431 2579 patch_prober.go:28] interesting pod/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Aug 04 19:44:53.666564 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:53.666500 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Aug 04 19:44:54.665497 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:54.665459 2579 patch_prober.go:28] interesting pod/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Aug 04 19:44:54.665898 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:54.665550 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Aug 04 19:44:55.665719 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:55.665675 2579 patch_prober.go:28] interesting pod/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Aug 04 19:44:55.666124 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:55.665747 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Aug 04 19:44:56.665179 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:56.665138 2579 patch_prober.go:28] interesting pod/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Aug 04 19:44:56.665397 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:56.665198 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Aug 04 19:44:57.665961 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:57.665923 2579 patch_prober.go:28] interesting pod/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Aug 04 19:44:57.666388 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:57.665997 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Aug 04 19:44:58.669726 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:58.669690 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:58.670103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:58.669994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:44:58.670855 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:44:58.670833 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:45:09.806826 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:09.806783 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc"] Aug 04 19:45:09.807249 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:09.807070 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" containerID="cri-o://edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7" gracePeriod=30 Aug 04 19:45:10.579832 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.579794 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f"] Aug 04 19:45:10.580216 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.580200 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" Aug 04 19:45:10.580282 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.580219 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" Aug 04 19:45:10.580282 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.580247 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" Aug 04 19:45:10.580282 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.580253 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" Aug 04 19:45:10.580397 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.580320 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" Aug 04 19:45:10.580397 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.580346 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac89d169-a83f-432e-bd4e-cd0bf0141331" containerName="istio-proxy" Aug 04 19:45:10.582610 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.582589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.587651 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.587624 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c53c7015-openshift-default-dockercfg-xt58n\"" Aug 04 19:45:10.606962 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.606922 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f"] Aug 04 19:45:10.706816 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.706771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-socket\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.706816 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.706819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sr64\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-kube-api-access-5sr64\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.706849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-certs\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.706870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda399b-7d57-4825-a08e-34354b300639-istio-podinfo\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.706956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-credential-socket\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.706982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda399b-7d57-4825-a08e-34354b300639-istiod-ca-cert\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.707027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-data\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.707057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-istio-token\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.707317 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.707123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-envoy\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808274 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-socket\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sr64\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-kube-api-access-5sr64\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-certs\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda399b-7d57-4825-a08e-34354b300639-istio-podinfo\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-credential-socket\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda399b-7d57-4825-a08e-34354b300639-istiod-ca-cert\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-data\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-istio-token\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-envoy\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.808803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-socket\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.809286 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-credential-socket\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.809286 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.808921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-certs\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.809286 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.809113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-data\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.809423 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.809314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda399b-7d57-4825-a08e-34354b300639-istiod-ca-cert\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.811031 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.811009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-envoy\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.811161 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.811124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda399b-7d57-4825-a08e-34354b300639-istio-podinfo\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.821351 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.821308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sr64\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-kube-api-access-5sr64\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.822005 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.821972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-istio-token\") pod \"e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:10.895851 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:10.895760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:11.045967 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.045934 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f"] Aug 04 19:45:11.048834 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:45:11.048799 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dda399b_7d57_4825_a08e_34354b300639.slice/crio-8a432c1592d3769d4821555dab25fd6e0458e11f896ab7f69850373279c61e5b WatchSource:0}: Error finding container 8a432c1592d3769d4821555dab25fd6e0458e11f896ab7f69850373279c61e5b: Status 404 returned error can't find the container with id 8a432c1592d3769d4821555dab25fd6e0458e11f896ab7f69850373279c61e5b Aug 04 19:45:11.051065 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.051030 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:11.051168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.051103 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:11.051168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.051133 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:11.342202 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.342162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" event={"ID":"2dda399b-7d57-4825-a08e-34354b300639","Type":"ContainerStarted","Data":"9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba"} Aug 04 19:45:11.342409 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.342264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" event={"ID":"2dda399b-7d57-4825-a08e-34354b300639","Type":"ContainerStarted","Data":"8a432c1592d3769d4821555dab25fd6e0458e11f896ab7f69850373279c61e5b"} Aug 04 19:45:11.365711 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.365651 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podStartSLOduration=1.365632412 podStartE2EDuration="1.365632412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:11.362799962 +0000 UTC m=+1788.303596708" watchObservedRunningTime="2026-08-04 19:45:11.365632412 +0000 UTC m=+1788.306429157" Aug 04 19:45:11.896620 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:11.896578 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:12.901584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:12.901549 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:13.352822 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:13.352786 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:13.353927 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:13.353901 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:13.672828 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:13.672738 2579 patch_prober.go:28] interesting pod/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:45:13.672971 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:13.672821 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:45:15.066946 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.066918 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:45:15.155944 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.155839 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-socket\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.155944 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.155894 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld6rb\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-kube-api-access-ld6rb\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.155944 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.155934 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-certs\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.155967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570af46d-2f6b-4eab-83ad-d80d90696ab8-istiod-ca-cert\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156018 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-credential-socket\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156089 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-token\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-podinfo\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156148 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:15.156236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-envoy\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-data\") pod \"570af46d-2f6b-4eab-83ad-d80d90696ab8\" (UID: \"570af46d-2f6b-4eab-83ad-d80d90696ab8\") " Aug 04 19:45:15.156645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:15.156645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156398 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:15.156645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570af46d-2f6b-4eab-83ad-d80d90696ab8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:15.156645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156615 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-data" (OuterVolumeSpecName: "istio-data") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:15.156900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156735 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.156900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156752 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.156900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156765 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.156900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156778 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570af46d-2f6b-4eab-83ad-d80d90696ab8-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.156900 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.156786 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.158344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.158304 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:45:15.158471 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.158453 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:15.158770 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.158742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-token" (OuterVolumeSpecName: "istio-token") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:15.158861 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.158741 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-kube-api-access-ld6rb" (OuterVolumeSpecName: "kube-api-access-ld6rb") pod "570af46d-2f6b-4eab-83ad-d80d90696ab8" (UID: "570af46d-2f6b-4eab-83ad-d80d90696ab8"). InnerVolumeSpecName "kube-api-access-ld6rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:15.257354 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.257293 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.257354 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.257356 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.257559 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.257366 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570af46d-2f6b-4eab-83ad-d80d90696ab8-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.257559 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.257375 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld6rb\" (UniqueName: \"kubernetes.io/projected/570af46d-2f6b-4eab-83ad-d80d90696ab8-kube-api-access-ld6rb\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:15.364520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.364437 2579 generic.go:356] "Generic (PLEG): container finished" podID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerID="edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7" exitCode=0 Aug 04 19:45:15.364520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.364493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" Aug 04 19:45:15.364767 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.364530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" event={"ID":"570af46d-2f6b-4eab-83ad-d80d90696ab8","Type":"ContainerDied","Data":"edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7"} Aug 04 19:45:15.364767 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.364602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc" event={"ID":"570af46d-2f6b-4eab-83ad-d80d90696ab8","Type":"ContainerDied","Data":"3e2ea4a61f3dd044f83d636aaa22838c50f4f9d517a4a45bf6078c5642b227d1"} Aug 04 19:45:15.364767 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.364632 2579 scope.go:117] "RemoveContainer" containerID="edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7" Aug 04 19:45:15.374276 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.374258 2579 scope.go:117] "RemoveContainer" containerID="c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a" Aug 04 19:45:15.383161 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.383136 2579 scope.go:117] "RemoveContainer" containerID="edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7" Aug 04 19:45:15.383553 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:45:15.383533 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7\": container with ID starting with edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7 not found: ID does not exist" containerID="edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7" Aug 04 19:45:15.383648 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.383565 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7"} err="failed to get container status \"edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7\": rpc error: code = NotFound desc = could not find container \"edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7\": container with ID starting with edf0700c40f1f60f657ab50ed877cd8f5b7be2c5f7fbc789c587f4d27ff024a7 not found: ID does not exist" Aug 04 19:45:15.383648 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.383590 2579 scope.go:117] "RemoveContainer" containerID="c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a" Aug 04 19:45:15.383925 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:45:15.383900 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a\": container with ID starting with c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a not found: ID does not exist" containerID="c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a" Aug 04 19:45:15.384016 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.383934 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a"} err="failed to get container status \"c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a\": rpc error: code = NotFound desc = could not find container \"c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a\": container with ID starting with c7bad32e9203cce84a593ad17245e7d63a07babbbf5702702c2295f5eb84442a not found: ID does not exist" Aug 04 19:45:15.386727 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.386702 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc"] Aug 04 19:45:15.393769 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.393588 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ead82904-gw-openshift-default-6fb488fdd8-cw4sc"] Aug 04 19:45:15.611557 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:15.611525 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" path="/var/lib/kubelet/pods/570af46d-2f6b-4eab-83ad-d80d90696ab8/volumes" Aug 04 19:45:23.661841 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:23.661814 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:45:23.663235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:23.663200 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:45:27.427591 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:27.427557 2579 generic.go:356] "Generic (PLEG): container finished" podID="2dda399b-7d57-4825-a08e-34354b300639" containerID="9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba" exitCode=0 Aug 04 19:45:27.428027 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:27.427627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" event={"ID":"2dda399b-7d57-4825-a08e-34354b300639","Type":"ContainerDied","Data":"9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba"} Aug 04 19:45:27.428072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:27.428035 2579 scope.go:117] "RemoveContainer" containerID="9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba" Aug 04 19:45:27.429084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:27.429060 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:27.429137 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:27.429125 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:27.429179 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:27.429160 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:28.353210 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:28.353169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:28.434823 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:28.434777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" event={"ID":"2dda399b-7d57-4825-a08e-34354b300639","Type":"ContainerStarted","Data":"3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e"} Aug 04 19:45:28.896239 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:28.896180 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:28.897524 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:28.897497 2579 patch_prober.go:28] interesting pod/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 04 19:45:28.897662 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:28.897548 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 04 19:45:29.896981 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:29.896945 2579 patch_prober.go:28] interesting pod/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 04 19:45:29.897435 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:29.897002 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 04 19:45:30.896729 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:30.896687 2579 patch_prober.go:28] interesting pod/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 04 19:45:30.896939 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:30.896748 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 04 19:45:31.897074 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:31.897033 2579 patch_prober.go:28] interesting pod/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 04 19:45:31.897552 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:31.897096 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 04 19:45:32.896718 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:32.896672 2579 patch_prober.go:28] interesting pod/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Aug 04 19:45:32.896897 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:32.896742 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Aug 04 19:45:33.900981 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:33.900948 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:33.901446 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:33.901255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:33.902096 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:33.902073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:45.673770 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:45.673730 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f"] Aug 04 19:45:45.674300 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:45.674011 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" containerID="cri-o://3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e" gracePeriod=30 Aug 04 19:45:46.400280 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.400240 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4"] Aug 04 19:45:46.400706 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.400683 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" Aug 04 19:45:46.400706 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.400702 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" Aug 04 19:45:46.400910 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.400720 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" Aug 04 19:45:46.400910 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.400726 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" Aug 04 19:45:46.400910 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.400796 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" Aug 04 19:45:46.402912 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.402891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.405172 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.405152 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3d6cfa2c-openshift-default-dockercfg-tpzq2\"" Aug 04 19:45:46.415216 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.415186 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4"] Aug 04 19:45:46.458244 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istiod-ca-cert\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-socket\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-certs\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-envoy\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-podinfo\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkgj\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-kube-api-access-2nkgj\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-credential-socket\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-token\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.458745 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.458600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-data\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559261 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-certs\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559261 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-envoy\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-podinfo\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkgj\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-kube-api-access-2nkgj\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-credential-socket\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-token\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-data\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istiod-ca-cert\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559879 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-socket\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.559879 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.559696 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-certs\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.560024 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.560002 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-data\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.560093 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.560059 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-credential-socket\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.560365 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.560342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-socket\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.560460 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.560437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istiod-ca-cert\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.561838 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.561815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-envoy\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.562039 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.562022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-podinfo\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.567935 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.567907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-token\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.568029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.567968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkgj\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-kube-api-access-2nkgj\") pod \"e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.714782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.714681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:46.874534 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.874498 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4"] Aug 04 19:45:46.875586 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:45:46.875557 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0569ce7b_6822_4f99_b9e1_5690b98dbb58.slice/crio-0ae21fe3fc3b82963f583fe9cac9079c49d36b250877cf817f7cfd4e4279065d WatchSource:0}: Error finding container 0ae21fe3fc3b82963f583fe9cac9079c49d36b250877cf817f7cfd4e4279065d: Status 404 returned error can't find the container with id 0ae21fe3fc3b82963f583fe9cac9079c49d36b250877cf817f7cfd4e4279065d Aug 04 19:45:46.878138 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.878106 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:46.878223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.878173 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:46.878223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:46.878203 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:45:47.545647 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:47.545609 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" event={"ID":"0569ce7b-6822-4f99-b9e1-5690b98dbb58","Type":"ContainerStarted","Data":"7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade"} Aug 04 19:45:47.545836 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:47.545692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" event={"ID":"0569ce7b-6822-4f99-b9e1-5690b98dbb58","Type":"ContainerStarted","Data":"0ae21fe3fc3b82963f583fe9cac9079c49d36b250877cf817f7cfd4e4279065d"} Aug 04 19:45:47.570573 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:47.570502 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podStartSLOduration=1.570477639 podStartE2EDuration="1.570477639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:45:47.567428862 +0000 UTC m=+1824.508225609" watchObservedRunningTime="2026-08-04 19:45:47.570477639 +0000 UTC m=+1824.511274387" Aug 04 19:45:47.715874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:47.715834 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:47.721293 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:47.721262 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:48.551928 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:48.551895 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:48.552998 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:48.552972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:45:48.904362 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:48.904262 2579 patch_prober.go:28] interesting pod/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:45:48.904719 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:48.904370 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:45:50.938505 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:50.938415 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:51.004106 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004053 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-data\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004106 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-envoy\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004398 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004145 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-credential-socket\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004398 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004197 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-socket\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004398 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004264 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-certs\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004398 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sr64\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-kube-api-access-5sr64\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004398 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda399b-7d57-4825-a08e-34354b300639-istio-podinfo\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004658 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-istio-token\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004658 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda399b-7d57-4825-a08e-34354b300639-istiod-ca-cert\") pod \"2dda399b-7d57-4825-a08e-34354b300639\" (UID: \"2dda399b-7d57-4825-a08e-34354b300639\") " Aug 04 19:45:51.004658 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004442 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:51.004795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004695 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:51.004795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:51.004904 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004798 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.004904 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004818 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.005006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004992 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:51.005056 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.004991 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dda399b-7d57-4825-a08e-34354b300639-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:45:51.005056 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.005025 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.006645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.006613 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:45:51.006922 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.006891 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dda399b-7d57-4825-a08e-34354b300639-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:45:51.006922 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.006899 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-kube-api-access-5sr64" (OuterVolumeSpecName: "kube-api-access-5sr64") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "kube-api-access-5sr64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:51.007563 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.007540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dda399b-7d57-4825-a08e-34354b300639" (UID: "2dda399b-7d57-4825-a08e-34354b300639"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:45:51.106377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.106343 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.106377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.106380 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda399b-7d57-4825-a08e-34354b300639-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.106589 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.106394 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.106589 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.106409 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda399b-7d57-4825-a08e-34354b300639-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.106589 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.106423 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sr64\" (UniqueName: \"kubernetes.io/projected/2dda399b-7d57-4825-a08e-34354b300639-kube-api-access-5sr64\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.106589 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.106440 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda399b-7d57-4825-a08e-34354b300639-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:45:51.568446 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.568403 2579 generic.go:356] "Generic (PLEG): container finished" podID="2dda399b-7d57-4825-a08e-34354b300639" containerID="3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e" exitCode=0 Aug 04 19:45:51.568604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.568456 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" Aug 04 19:45:51.568604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.568470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" event={"ID":"2dda399b-7d57-4825-a08e-34354b300639","Type":"ContainerDied","Data":"3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e"} Aug 04 19:45:51.568604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.568530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f" event={"ID":"2dda399b-7d57-4825-a08e-34354b300639","Type":"ContainerDied","Data":"8a432c1592d3769d4821555dab25fd6e0458e11f896ab7f69850373279c61e5b"} Aug 04 19:45:51.568604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.568561 2579 scope.go:117] "RemoveContainer" containerID="3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e" Aug 04 19:45:51.578402 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.578380 2579 scope.go:117] "RemoveContainer" containerID="9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba" Aug 04 19:45:51.586892 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.586870 2579 scope.go:117] "RemoveContainer" containerID="3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e" Aug 04 19:45:51.587221 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:45:51.587201 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e\": container with ID starting with 3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e not found: ID does not exist" containerID="3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e" Aug 04 19:45:51.587272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.587234 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e"} err="failed to get container status \"3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e\": rpc error: code = NotFound desc = could not find container \"3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e\": container with ID starting with 3ab3a71cac200daeb1ef64ef1950f359c732c07f33e9c72b16a28a8043ac1d1e not found: ID does not exist" Aug 04 19:45:51.587272 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.587254 2579 scope.go:117] "RemoveContainer" containerID="9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba" Aug 04 19:45:51.587522 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:45:51.587503 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba\": container with ID starting with 9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba not found: ID does not exist" containerID="9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba" Aug 04 19:45:51.587574 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.587530 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba"} err="failed to get container status \"9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba\": rpc error: code = NotFound desc = could not find container \"9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba\": container with ID starting with 9664758e5e50775f91f61cd286f2cc6c304843b16ff50ca0761cc058e85ddeba not found: ID does not exist" Aug 04 19:45:51.591457 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.591427 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f"] Aug 04 19:45:51.597385 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.597344 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c53c7015-openshift-default-6d99896fbb-c668f"] Aug 04 19:45:51.613700 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:45:51.613663 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dda399b-7d57-4825-a08e-34354b300639" path="/var/lib/kubelet/pods/2dda399b-7d57-4825-a08e-34354b300639/volumes" Aug 04 19:46:03.553470 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.553434 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:03.553902 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.553493 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:03.630949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.630913 2579 generic.go:356] "Generic (PLEG): container finished" podID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerID="7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade" exitCode=0 Aug 04 19:46:03.631125 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.630957 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" event={"ID":"0569ce7b-6822-4f99-b9e1-5690b98dbb58","Type":"ContainerDied","Data":"7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade"} Aug 04 19:46:03.631411 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.631395 2579 scope.go:117] "RemoveContainer" containerID="7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade" Aug 04 19:46:03.632450 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.632414 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:46:03.632570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.632501 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:46:03.632570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:03.632534 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:46:04.649401 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:04.649353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" event={"ID":"0569ce7b-6822-4f99-b9e1-5690b98dbb58","Type":"ContainerStarted","Data":"7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482"} Aug 04 19:46:04.715835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:04.715773 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:46:04.716139 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:04.716118 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:04.716195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:04.716178 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:05.715238 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:05.715201 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:05.715764 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:05.715267 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:06.715235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:06.715195 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:06.715670 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:06.715268 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:07.715805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:07.715765 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:07.716237 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:07.715823 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:08.715720 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:08.715681 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:08.715910 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:08.715739 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:09.716103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:09.716058 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Aug 04 19:46:09.716538 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:09.716137 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Aug 04 19:46:10.719559 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:10.719524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:46:10.719965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:10.719922 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:46:10.720638 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:10.720616 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:46:21.026397 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:21.026357 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4"] Aug 04 19:46:21.026913 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:21.026589 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" containerID="cri-o://7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482" gracePeriod=30 Aug 04 19:46:25.722250 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:25.722206 2579 patch_prober.go:28] interesting pod/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:46:25.722636 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:25.722306 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:46:26.286834 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.286805 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:46:26.347994 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.347902 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-socket\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.347994 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.347965 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-envoy\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.347994 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.347993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istiod-ca-cert\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348043 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-podinfo\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-data\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348098 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nkgj\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-kube-api-access-2nkgj\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-token\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348146 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-certs\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-credential-socket\") pod \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\" (UID: \"0569ce7b-6822-4f99-b9e1-5690b98dbb58\") " Aug 04 19:46:26.348288 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348197 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:46:26.348636 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348436 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-data" (OuterVolumeSpecName: "istio-data") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:46:26.348636 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:46:26.348636 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348523 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.348636 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348590 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:46:26.348838 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.348727 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:46:26.350444 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.350400 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-kube-api-access-2nkgj" (OuterVolumeSpecName: "kube-api-access-2nkgj") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "kube-api-access-2nkgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:46:26.350563 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.350470 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:46:26.350563 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.350494 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-token" (OuterVolumeSpecName: "istio-token") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:46:26.350563 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.350494 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0569ce7b-6822-4f99-b9e1-5690b98dbb58" (UID: "0569ce7b-6822-4f99-b9e1-5690b98dbb58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:46:26.450076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450037 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450069 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450080 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450076 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450090 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450380 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450097 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450380 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450105 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nkgj\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-kube-api-access-2nkgj\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450380 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450114 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0569ce7b-6822-4f99-b9e1-5690b98dbb58-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.450380 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.450121 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0569ce7b-6822-4f99-b9e1-5690b98dbb58-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:46:26.760724 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.760633 2579 generic.go:356] "Generic (PLEG): container finished" podID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerID="7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482" exitCode=0 Aug 04 19:46:26.760724 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.760683 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" Aug 04 19:46:26.761152 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.760727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" event={"ID":"0569ce7b-6822-4f99-b9e1-5690b98dbb58","Type":"ContainerDied","Data":"7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482"} Aug 04 19:46:26.761152 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.760788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4" event={"ID":"0569ce7b-6822-4f99-b9e1-5690b98dbb58","Type":"ContainerDied","Data":"0ae21fe3fc3b82963f583fe9cac9079c49d36b250877cf817f7cfd4e4279065d"} Aug 04 19:46:26.761152 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.760814 2579 scope.go:117] "RemoveContainer" containerID="7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482" Aug 04 19:46:26.770416 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.770394 2579 scope.go:117] "RemoveContainer" containerID="7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade" Aug 04 19:46:26.778916 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.778895 2579 scope.go:117] "RemoveContainer" containerID="7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482" Aug 04 19:46:26.779212 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:46:26.779192 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482\": container with ID starting with 7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482 not found: ID does not exist" containerID="7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482" Aug 04 19:46:26.779277 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.779221 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482"} err="failed to get container status \"7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482\": rpc error: code = NotFound desc = could not find container \"7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482\": container with ID starting with 7d4fb61e7aec5220317c2b56115368ff08daa21110c551a49563918a0ee4c482 not found: ID does not exist" Aug 04 19:46:26.779277 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.779240 2579 scope.go:117] "RemoveContainer" containerID="7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade" Aug 04 19:46:26.779501 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:46:26.779482 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade\": container with ID starting with 7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade not found: ID does not exist" containerID="7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade" Aug 04 19:46:26.779560 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.779507 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade"} err="failed to get container status \"7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade\": rpc error: code = NotFound desc = could not find container \"7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade\": container with ID starting with 7462b760332f281f4d1f1191046fd2d9a1d76470177ee561356eb3bc545a6ade not found: ID does not exist" Aug 04 19:46:26.782466 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.782441 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4"] Aug 04 19:46:26.786433 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:26.786410 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3d6cfa2c-openshift-default-55668679b-mtgr4"] Aug 04 19:46:27.611007 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:46:27.610973 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" path="/var/lib/kubelet/pods/0569ce7b-6822-4f99-b9e1-5690b98dbb58/volumes" Aug 04 19:48:19.553195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553102 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp"] Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553524 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553538 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553548 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553554 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553563 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553569 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553639 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553649 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553660 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553666 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dda399b-7d57-4825-a08e-34354b300639" containerName="istio-proxy" Aug 04 19:48:19.553795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.553673 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="570af46d-2f6b-4eab-83ad-d80d90696ab8" containerName="istio-proxy" Aug 04 19:48:19.555835 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.555816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.558173 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.558153 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8129f458-openshift-default-dockercfg-t99lr\"" Aug 04 19:48:19.584963 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.584919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp"] Aug 04 19:48:19.704124 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-data\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704313 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-token\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704313 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223d998-7aa6-4f63-9281-f0515f8a1067-istiod-ca-cert\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704313 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-credential-socket\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-socket\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-envoy\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rlc\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-kube-api-access-l7rlc\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704575 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-certs\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.704688 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.704665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-podinfo\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.805895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-socket\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.805941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-envoy\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rlc\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-kube-api-access-l7rlc\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-certs\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806234 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-podinfo\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806436 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-data\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806436 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-token\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806436 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223d998-7aa6-4f63-9281-f0515f8a1067-istiod-ca-cert\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806436 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-credential-socket\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806641 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-certs\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806641 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806587 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-socket\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-data\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.806749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.806717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-credential-socket\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.807111 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.807089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223d998-7aa6-4f63-9281-f0515f8a1067-istiod-ca-cert\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.808409 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.808386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-envoy\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.808795 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.808777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-podinfo\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.816250 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.816201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-token\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.816537 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.816514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rlc\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-kube-api-access-l7rlc\") pod \"e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:19.866317 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:19.866279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:20.010827 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.010730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp"] Aug 04 19:48:20.013879 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:48:20.013844 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2223d998_7aa6_4f63_9281_f0515f8a1067.slice/crio-1eede224f3195d3c1673676253ec9471f9387e00cdc84b2f67ca41c8d54bae93 WatchSource:0}: Error finding container 1eede224f3195d3c1673676253ec9471f9387e00cdc84b2f67ca41c8d54bae93: Status 404 returned error can't find the container with id 1eede224f3195d3c1673676253ec9471f9387e00cdc84b2f67ca41c8d54bae93 Aug 04 19:48:20.016586 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.016540 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:48:20.016660 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.016648 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:48:20.016710 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.016693 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:48:20.306479 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.306437 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" event={"ID":"2223d998-7aa6-4f63-9281-f0515f8a1067","Type":"ContainerStarted","Data":"78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f"} Aug 04 19:48:20.306724 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.306492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" event={"ID":"2223d998-7aa6-4f63-9281-f0515f8a1067","Type":"ContainerStarted","Data":"1eede224f3195d3c1673676253ec9471f9387e00cdc84b2f67ca41c8d54bae93"} Aug 04 19:48:20.328188 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.328132 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podStartSLOduration=1.328114269 podStartE2EDuration="1.328114269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:48:20.326905169 +0000 UTC m=+1977.267701916" watchObservedRunningTime="2026-08-04 19:48:20.328114269 +0000 UTC m=+1977.268911015" Aug 04 19:48:20.866794 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:20.866756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:21.872268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:21.872234 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:22.316107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:22.316076 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:22.317204 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:22.317183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:36.395254 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:36.395215 2579 generic.go:356] "Generic (PLEG): container finished" podID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerID="78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f" exitCode=0 Aug 04 19:48:36.395817 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:36.395281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" event={"ID":"2223d998-7aa6-4f63-9281-f0515f8a1067","Type":"ContainerDied","Data":"78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f"} Aug 04 19:48:36.395887 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:36.395833 2579 scope.go:117] "RemoveContainer" containerID="78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f" Aug 04 19:48:36.396931 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:36.396902 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:48:36.397034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:36.396987 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:48:36.397102 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:36.397034 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:48:37.316716 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:37.316661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:37.401924 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:37.401888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" event={"ID":"2223d998-7aa6-4f63-9281-f0515f8a1067","Type":"ContainerStarted","Data":"457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af"} Aug 04 19:48:37.867582 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:37.867542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:37.868774 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:37.868743 2579 patch_prober.go:28] interesting pod/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 04 19:48:37.868902 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:37.868814 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 04 19:48:38.866810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:38.866766 2579 patch_prober.go:28] interesting pod/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 04 19:48:38.867365 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:38.866841 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 04 19:48:39.867665 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:39.867626 2579 patch_prober.go:28] interesting pod/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 04 19:48:39.868160 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:39.867697 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 04 19:48:40.867361 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:40.867288 2579 patch_prober.go:28] interesting pod/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 04 19:48:40.867587 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:40.867388 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 04 19:48:41.867109 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:41.867069 2579 patch_prober.go:28] interesting pod/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 04 19:48:41.867553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:41.867139 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 04 19:48:42.871050 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:42.871011 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:42.871538 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:42.871306 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:48:42.872058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:48:42.872042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:49:12.257475 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:12.257430 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp"] Aug 04 19:49:12.258015 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:12.257776 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" containerID="cri-o://457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af" gracePeriod=30 Aug 04 19:49:12.874593 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:12.874555 2579 patch_prober.go:28] interesting pod/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:49:12.874773 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:12.874616 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:49:17.517281 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.517250 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:49:17.549315 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549285 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-podinfo\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549349 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223d998-7aa6-4f63-9281-f0515f8a1067-istiod-ca-cert\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-socket\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-envoy\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549424 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-data\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-certs\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-credential-socket\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549570 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549538 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7rlc\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-kube-api-access-l7rlc\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549602 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-token\") pod \"2223d998-7aa6-4f63-9281-f0515f8a1067\" (UID: \"2223d998-7aa6-4f63-9281-f0515f8a1067\") " Aug 04 19:49:17.549992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:49:17.549992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2223d998-7aa6-4f63-9281-f0515f8a1067-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:49:17.549992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.549944 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.550264 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.550201 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:49:17.550420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.550274 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:49:17.550420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.550371 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-data" (OuterVolumeSpecName: "istio-data") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:49:17.552321 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.552290 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:49:17.552443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.552373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:49:17.552443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.552380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-kube-api-access-l7rlc" (OuterVolumeSpecName: "kube-api-access-l7rlc") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "kube-api-access-l7rlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:49:17.552443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.552393 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-token" (OuterVolumeSpecName: "istio-token") pod "2223d998-7aa6-4f63-9281-f0515f8a1067" (UID: "2223d998-7aa6-4f63-9281-f0515f8a1067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:49:17.601902 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.601868 2579 generic.go:356] "Generic (PLEG): container finished" podID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerID="457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af" exitCode=0 Aug 04 19:49:17.602105 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.601918 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" Aug 04 19:49:17.602105 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.601939 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" event={"ID":"2223d998-7aa6-4f63-9281-f0515f8a1067","Type":"ContainerDied","Data":"457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af"} Aug 04 19:49:17.602105 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.601992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp" event={"ID":"2223d998-7aa6-4f63-9281-f0515f8a1067","Type":"ContainerDied","Data":"1eede224f3195d3c1673676253ec9471f9387e00cdc84b2f67ca41c8d54bae93"} Aug 04 19:49:17.602105 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.602021 2579 scope.go:117] "RemoveContainer" containerID="457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af" Aug 04 19:49:17.614264 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.614239 2579 scope.go:117] "RemoveContainer" containerID="78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f" Aug 04 19:49:17.623068 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.623043 2579 scope.go:117] "RemoveContainer" containerID="457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af" Aug 04 19:49:17.623497 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:49:17.623461 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af\": container with ID starting with 457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af not found: ID does not exist" containerID="457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af" Aug 04 19:49:17.623658 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.623504 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af"} err="failed to get container status \"457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af\": rpc error: code = NotFound desc = could not find container \"457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af\": container with ID starting with 457bee5171335096ca66ce1cc2c59eaf0c457abd093098d8716b91b1bb67f0af not found: ID does not exist" Aug 04 19:49:17.623658 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.623534 2579 scope.go:117] "RemoveContainer" containerID="78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f" Aug 04 19:49:17.623950 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:49:17.623904 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f\": container with ID starting with 78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f not found: ID does not exist" containerID="78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f" Aug 04 19:49:17.624162 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.623945 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f"} err="failed to get container status \"78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f\": rpc error: code = NotFound desc = could not find container \"78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f\": container with ID starting with 78eaf26ccf995a00b810b5fa34513e9e2cc7afbd4a9c4c28bbe60c34dd6c006f not found: ID does not exist" Aug 04 19:49:17.626002 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.625977 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp"] Aug 04 19:49:17.629964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.629938 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8129f458-openshift-default-7b984bcc96-wlmcp"] Aug 04 19:49:17.651320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651291 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651319 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2223d998-7aa6-4f63-9281-f0515f8a1067-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651341 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651350 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651572 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651359 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651572 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651367 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2223d998-7aa6-4f63-9281-f0515f8a1067-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651572 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651377 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rlc\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-kube-api-access-l7rlc\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:17.651572 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:17.651387 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2223d998-7aa6-4f63-9281-f0515f8a1067-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:19.610561 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:19.610517 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" path="/var/lib/kubelet/pods/2223d998-7aa6-4f63-9281-f0515f8a1067/volumes" Aug 04 19:49:53.325656 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.325559 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj"] Aug 04 19:49:53.326236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.325827 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" podUID="8b78b673-4f3d-4caa-8c2b-f7025bd2a930" containerName="manager" containerID="cri-o://c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140" gracePeriod=10 Aug 04 19:49:53.586390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.586262 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:49:53.693637 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.693593 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28h6p\" (UniqueName: \"kubernetes.io/projected/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-kube-api-access-28h6p\") pod \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " Aug 04 19:49:53.693818 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.693681 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-extensions-socket-volume\") pod \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\" (UID: \"8b78b673-4f3d-4caa-8c2b-f7025bd2a930\") " Aug 04 19:49:53.694103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.694072 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8b78b673-4f3d-4caa-8c2b-f7025bd2a930" (UID: "8b78b673-4f3d-4caa-8c2b-f7025bd2a930"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:49:53.695901 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.695870 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-kube-api-access-28h6p" (OuterVolumeSpecName: "kube-api-access-28h6p") pod "8b78b673-4f3d-4caa-8c2b-f7025bd2a930" (UID: "8b78b673-4f3d-4caa-8c2b-f7025bd2a930"). InnerVolumeSpecName "kube-api-access-28h6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:49:53.774044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.774011 2579 generic.go:356] "Generic (PLEG): container finished" podID="8b78b673-4f3d-4caa-8c2b-f7025bd2a930" containerID="c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140" exitCode=0 Aug 04 19:49:53.774223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.774059 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" Aug 04 19:49:53.774223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.774069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" event={"ID":"8b78b673-4f3d-4caa-8c2b-f7025bd2a930","Type":"ContainerDied","Data":"c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140"} Aug 04 19:49:53.774223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.774117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj" event={"ID":"8b78b673-4f3d-4caa-8c2b-f7025bd2a930","Type":"ContainerDied","Data":"712a63107513aad2053168ce4992a83e8359fb7c59f6021337c3b1fa6af47e60"} Aug 04 19:49:53.774223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.774139 2579 scope.go:117] "RemoveContainer" containerID="c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140" Aug 04 19:49:53.785287 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.785266 2579 scope.go:117] "RemoveContainer" containerID="c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140" Aug 04 19:49:53.785624 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:49:53.785605 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140\": container with ID starting with c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140 not found: ID does not exist" containerID="c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140" Aug 04 19:49:53.785685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.785635 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140"} err="failed to get container status \"c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140\": rpc error: code = NotFound desc = could not find container \"c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140\": container with ID starting with c61cd105d95a1afaaa111cba9ca718cda8a3422636dc83ad2be7f1105a2c0140 not found: ID does not exist" Aug 04 19:49:53.795222 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.795189 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-extensions-socket-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:53.795222 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.795221 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h6p\" (UniqueName: \"kubernetes.io/projected/8b78b673-4f3d-4caa-8c2b-f7025bd2a930-kube-api-access-28h6p\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:49:53.797141 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.797111 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj"] Aug 04 19:49:53.800510 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:53.800483 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4ftnj"] Aug 04 19:49:55.610188 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:49:55.610149 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b78b673-4f3d-4caa-8c2b-f7025bd2a930" path="/var/lib/kubelet/pods/8b78b673-4f3d-4caa-8c2b-f7025bd2a930/volumes" Aug 04 19:50:23.692322 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:50:23.692276 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:50:23.694475 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:50:23.694441 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:51:08.949600 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949558 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx"] Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949939 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949951 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949959 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949964 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949972 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b78b673-4f3d-4caa-8c2b-f7025bd2a930" containerName="manager" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949977 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b78b673-4f3d-4caa-8c2b-f7025bd2a930" containerName="manager" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949984 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.949990 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0569ce7b-6822-4f99-b9e1-5690b98dbb58" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.950048 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b78b673-4f3d-4caa-8c2b-f7025bd2a930" containerName="manager" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.950059 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" Aug 04 19:51:08.950103 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.950068 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2223d998-7aa6-4f63-9281-f0515f8a1067" containerName="istio-proxy" Aug 04 19:51:08.953043 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.953021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:08.955595 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.955566 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-846sv\"" Aug 04 19:51:08.965760 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:08.965730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx"] Aug 04 19:51:09.085268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.085225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglf9\" (UniqueName: \"kubernetes.io/projected/71660ffb-f55f-457d-8fad-fc37bb9fd937-kube-api-access-pglf9\") pod \"kuadrant-operator-controller-manager-68cf589589-gfxkx\" (UID: \"71660ffb-f55f-457d-8fad-fc37bb9fd937\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.085488 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.085385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71660ffb-f55f-457d-8fad-fc37bb9fd937-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-gfxkx\" (UID: \"71660ffb-f55f-457d-8fad-fc37bb9fd937\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.186249 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.186207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglf9\" (UniqueName: \"kubernetes.io/projected/71660ffb-f55f-457d-8fad-fc37bb9fd937-kube-api-access-pglf9\") pod \"kuadrant-operator-controller-manager-68cf589589-gfxkx\" (UID: \"71660ffb-f55f-457d-8fad-fc37bb9fd937\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.186499 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.186287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71660ffb-f55f-457d-8fad-fc37bb9fd937-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-gfxkx\" (UID: \"71660ffb-f55f-457d-8fad-fc37bb9fd937\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.186668 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.186648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/71660ffb-f55f-457d-8fad-fc37bb9fd937-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-gfxkx\" (UID: \"71660ffb-f55f-457d-8fad-fc37bb9fd937\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.194969 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.194937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglf9\" (UniqueName: \"kubernetes.io/projected/71660ffb-f55f-457d-8fad-fc37bb9fd937-kube-api-access-pglf9\") pod \"kuadrant-operator-controller-manager-68cf589589-gfxkx\" (UID: \"71660ffb-f55f-457d-8fad-fc37bb9fd937\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.263419 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.263303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:09.416923 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:09.416894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx"] Aug 04 19:51:09.418444 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:51:09.418417 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71660ffb_f55f_457d_8fad_fc37bb9fd937.slice/crio-55ff4373da840a3ae6ca62e244ca2e93953b479794c2f9e0ce5978b8fe6c14e2 WatchSource:0}: Error finding container 55ff4373da840a3ae6ca62e244ca2e93953b479794c2f9e0ce5978b8fe6c14e2: Status 404 returned error can't find the container with id 55ff4373da840a3ae6ca62e244ca2e93953b479794c2f9e0ce5978b8fe6c14e2 Aug 04 19:51:10.131508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:10.131441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" event={"ID":"71660ffb-f55f-457d-8fad-fc37bb9fd937","Type":"ContainerStarted","Data":"3df371c1b2be6172658fc7a7fabca93078984edbb2f52c99864595442bdbb1e1"} Aug 04 19:51:10.131508 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:10.131506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" event={"ID":"71660ffb-f55f-457d-8fad-fc37bb9fd937","Type":"ContainerStarted","Data":"55ff4373da840a3ae6ca62e244ca2e93953b479794c2f9e0ce5978b8fe6c14e2"} Aug 04 19:51:10.132054 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:10.131763 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:10.159156 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:10.159092 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" podStartSLOduration=2.159073998 podStartE2EDuration="2.159073998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:51:10.157004453 +0000 UTC m=+2147.097801199" watchObservedRunningTime="2026-08-04 19:51:10.159073998 +0000 UTC m=+2147.099870743" Aug 04 19:51:21.139037 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:21.138944 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-gfxkx" Aug 04 19:51:27.985836 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:27.985799 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l"] Aug 04 19:51:27.989949 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:27.989924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:27.992375 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:27.992341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fb61cc13-openshift-default-dockercfg-sfmlh\"" Aug 04 19:51:28.001884 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.001850 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l"] Aug 04 19:51:28.047216 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvc94\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-kube-api-access-qvc94\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047216 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-credential-socket\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-podinfo\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-data\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9a20f-efe5-4ffc-95f2-4df025516802-istiod-ca-cert\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-certs\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-envoy\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-socket\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.047826 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.047528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-token\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.148974 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.148932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc94\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-kube-api-access-qvc94\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149160 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.148999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-credential-socket\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149160 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-podinfo\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149160 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-data\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9a20f-efe5-4ffc-95f2-4df025516802-istiod-ca-cert\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-certs\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-envoy\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-socket\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149377 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-token\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149678 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-credential-socket\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149678 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-data\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149678 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-certs\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149850 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-socket\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.149980 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.149956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9a20f-efe5-4ffc-95f2-4df025516802-istiod-ca-cert\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.151580 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.151561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-envoy\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.151699 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.151682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-podinfo\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.157280 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.157248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-token\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.157423 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.157409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvc94\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-kube-api-access-qvc94\") pod \"e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.302255 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.302215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:28.451976 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.451945 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l"] Aug 04 19:51:28.454457 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:51:28.454408 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef9a20f_efe5_4ffc_95f2_4df025516802.slice/crio-bd1b365d792076cb99b18262d21190922cf4de89c57e08360dbb1413af39343d WatchSource:0}: Error finding container bd1b365d792076cb99b18262d21190922cf4de89c57e08360dbb1413af39343d: Status 404 returned error can't find the container with id bd1b365d792076cb99b18262d21190922cf4de89c57e08360dbb1413af39343d Aug 04 19:51:28.457058 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.457022 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:51:28.457146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.457092 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:51:28.457146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:28.457119 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:51:29.222938 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:29.222897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" event={"ID":"0ef9a20f-efe5-4ffc-95f2-4df025516802","Type":"ContainerStarted","Data":"bff54a6221d9aba8f61cd3e5e67ffca93ef482865868a0d184bb0b1eb0b693e9"} Aug 04 19:51:29.223380 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:29.222959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" event={"ID":"0ef9a20f-efe5-4ffc-95f2-4df025516802","Type":"ContainerStarted","Data":"bd1b365d792076cb99b18262d21190922cf4de89c57e08360dbb1413af39343d"} Aug 04 19:51:29.244778 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:29.244722 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podStartSLOduration=2.244703907 podStartE2EDuration="2.244703907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:51:29.242850959 +0000 UTC m=+2166.183647707" watchObservedRunningTime="2026-08-04 19:51:29.244703907 +0000 UTC m=+2166.185500654" Aug 04 19:51:29.302927 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:29.302887 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:29.308118 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:29.308085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:30.230734 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:30.230644 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:30.235212 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:30.235183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:44.309648 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:44.309606 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerID="bff54a6221d9aba8f61cd3e5e67ffca93ef482865868a0d184bb0b1eb0b693e9" exitCode=0 Aug 04 19:51:44.310246 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:44.309678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" event={"ID":"0ef9a20f-efe5-4ffc-95f2-4df025516802","Type":"ContainerDied","Data":"bff54a6221d9aba8f61cd3e5e67ffca93ef482865868a0d184bb0b1eb0b693e9"} Aug 04 19:51:44.310246 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:44.310108 2579 scope.go:117] "RemoveContainer" containerID="bff54a6221d9aba8f61cd3e5e67ffca93ef482865868a0d184bb0b1eb0b693e9" Aug 04 19:51:44.311243 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:44.311216 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:51:44.311300 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:44.311285 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:51:44.311366 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:44.311320 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:51:45.230919 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:45.230875 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:45.317767 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:45.317734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" event={"ID":"0ef9a20f-efe5-4ffc-95f2-4df025516802","Type":"ContainerStarted","Data":"80e9f715742e77593875a86052390e4326371e35108b4edead13acd231060a18"} Aug 04 19:51:46.303409 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:46.303367 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:46.304571 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:46.304543 2579 patch_prober.go:28] interesting pod/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Aug 04 19:51:46.304697 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:46.304594 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Aug 04 19:51:47.302798 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:47.302757 2579 patch_prober.go:28] interesting pod/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Aug 04 19:51:47.303195 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:47.302821 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Aug 04 19:51:48.303229 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:48.303191 2579 patch_prober.go:28] interesting pod/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Aug 04 19:51:48.303660 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:48.303256 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Aug 04 19:51:49.303063 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:49.303027 2579 patch_prober.go:28] interesting pod/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Aug 04 19:51:49.303248 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:49.303090 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Aug 04 19:51:50.303474 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:50.303427 2579 patch_prober.go:28] interesting pod/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Aug 04 19:51:50.304028 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:50.303506 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Aug 04 19:51:51.307506 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:51.307472 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:51.307886 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:51.307729 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:51:51.308546 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:51:51.308524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:52:11.990320 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:11.990285 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds"] Aug 04 19:52:11.993725 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:11.993698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:11.996121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:11.996083 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb61cc13\"/\"kube-root-ca.crt\"" Aug 04 19:52:11.996121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:11.996083 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb61cc13\"/\"openshift-service-ca.crt\"" Aug 04 19:52:11.996992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:11.996966 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb61cc13\"/\"e2e-lifecycle-model-fb61cc13-kserve-self-signed-certs\"" Aug 04 19:52:11.997105 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:11.997013 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb61cc13\"/\"default-dockercfg-26rsr\"" Aug 04 19:52:12.002647 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.002619 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds"] Aug 04 19:52:12.076629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.076591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-dshm\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.076844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.076675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-model-cache\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.076844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.076701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jjd\" (UniqueName: \"kubernetes.io/projected/16415033-f7ab-483c-997b-91ff1bdd64a7-kube-api-access-s8jjd\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.076844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.076737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-home\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.076844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.076766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16415033-f7ab-483c-997b-91ff1bdd64a7-tls-certs\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.076844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.076797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-tmp-dir\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.177878 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.177835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-dshm\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.177904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-model-cache\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.177931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jjd\" (UniqueName: \"kubernetes.io/projected/16415033-f7ab-483c-997b-91ff1bdd64a7-kube-api-access-s8jjd\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.177964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-home\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.177996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16415033-f7ab-483c-997b-91ff1bdd64a7-tls-certs\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.178027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-tmp-dir\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.178403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-home\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.178486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-tmp-dir\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.178629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.178562 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-model-cache\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.181414 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.180965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-dshm\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.181414 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.181045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16415033-f7ab-483c-997b-91ff1bdd64a7-tls-certs\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.187499 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.187466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jjd\" (UniqueName: \"kubernetes.io/projected/16415033-f7ab-483c-997b-91ff1bdd64a7-kube-api-access-s8jjd\") pod \"e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.305415 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.305367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:12.462608 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:12.462580 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds"] Aug 04 19:52:12.465360 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:52:12.465315 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16415033_f7ab_483c_997b_91ff1bdd64a7.slice/crio-e19b70f1a3d3501e4148917c67bf7eb8d54de90581842d7f1f5095eba5346e94 WatchSource:0}: Error finding container e19b70f1a3d3501e4148917c67bf7eb8d54de90581842d7f1f5095eba5346e94: Status 404 returned error can't find the container with id e19b70f1a3d3501e4148917c67bf7eb8d54de90581842d7f1f5095eba5346e94 Aug 04 19:52:13.464811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:13.464774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" event={"ID":"16415033-f7ab-483c-997b-91ff1bdd64a7","Type":"ContainerStarted","Data":"d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e"} Aug 04 19:52:13.464811 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:13.464819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" event={"ID":"16415033-f7ab-483c-997b-91ff1bdd64a7","Type":"ContainerStarted","Data":"e19b70f1a3d3501e4148917c67bf7eb8d54de90581842d7f1f5095eba5346e94"} Aug 04 19:52:13.481595 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:13.481538 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" podStartSLOduration=2.481520343 podStartE2EDuration="2.481520343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:52:13.48067326 +0000 UTC m=+2210.421470040" watchObservedRunningTime="2026-08-04 19:52:13.481520343 +0000 UTC m=+2210.422317089" Aug 04 19:52:22.305676 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:22.305636 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:22.305676 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:22.305687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:22.318545 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:22.318507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:52:22.519455 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:52:22.519422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:53:04.707829 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:04.707748 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds"] Aug 04 19:53:19.727863 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:19.727813 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" podUID="16415033-f7ab-483c-997b-91ff1bdd64a7" containerName="main" containerID="cri-o://d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e" gracePeriod=45 Aug 04 19:53:19.991621 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:19.991554 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:53:20.033414 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033382 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-model-cache\") pod \"16415033-f7ab-483c-997b-91ff1bdd64a7\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " Aug 04 19:53:20.033594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033433 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16415033-f7ab-483c-997b-91ff1bdd64a7-tls-certs\") pod \"16415033-f7ab-483c-997b-91ff1bdd64a7\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " Aug 04 19:53:20.033594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-dshm\") pod \"16415033-f7ab-483c-997b-91ff1bdd64a7\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " Aug 04 19:53:20.033594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033482 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8jjd\" (UniqueName: \"kubernetes.io/projected/16415033-f7ab-483c-997b-91ff1bdd64a7-kube-api-access-s8jjd\") pod \"16415033-f7ab-483c-997b-91ff1bdd64a7\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " Aug 04 19:53:20.033594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-tmp-dir\") pod \"16415033-f7ab-483c-997b-91ff1bdd64a7\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " Aug 04 19:53:20.033594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033552 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-home\") pod \"16415033-f7ab-483c-997b-91ff1bdd64a7\" (UID: \"16415033-f7ab-483c-997b-91ff1bdd64a7\") " Aug 04 19:53:20.034082 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.033884 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16415033-f7ab-483c-997b-91ff1bdd64a7" (UID: "16415033-f7ab-483c-997b-91ff1bdd64a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:20.034082 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.034015 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "16415033-f7ab-483c-997b-91ff1bdd64a7" (UID: "16415033-f7ab-483c-997b-91ff1bdd64a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:20.034226 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.034112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-home" (OuterVolumeSpecName: "home") pod "16415033-f7ab-483c-997b-91ff1bdd64a7" (UID: "16415033-f7ab-483c-997b-91ff1bdd64a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:20.036005 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.035975 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-dshm" (OuterVolumeSpecName: "dshm") pod "16415033-f7ab-483c-997b-91ff1bdd64a7" (UID: "16415033-f7ab-483c-997b-91ff1bdd64a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:20.036142 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.036116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16415033-f7ab-483c-997b-91ff1bdd64a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16415033-f7ab-483c-997b-91ff1bdd64a7" (UID: "16415033-f7ab-483c-997b-91ff1bdd64a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:53:20.036198 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.036166 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16415033-f7ab-483c-997b-91ff1bdd64a7-kube-api-access-s8jjd" (OuterVolumeSpecName: "kube-api-access-s8jjd") pod "16415033-f7ab-483c-997b-91ff1bdd64a7" (UID: "16415033-f7ab-483c-997b-91ff1bdd64a7"). InnerVolumeSpecName "kube-api-access-s8jjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:53:20.134705 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.134663 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16415033-f7ab-483c-997b-91ff1bdd64a7-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:20.134705 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.134694 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:20.134705 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.134707 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jjd\" (UniqueName: \"kubernetes.io/projected/16415033-f7ab-483c-997b-91ff1bdd64a7-kube-api-access-s8jjd\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:20.134965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.134721 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:20.134965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.134730 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:20.134965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.134740 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16415033-f7ab-483c-997b-91ff1bdd64a7-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:20.807146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.807111 2579 generic.go:356] "Generic (PLEG): container finished" podID="16415033-f7ab-483c-997b-91ff1bdd64a7" containerID="d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e" exitCode=0 Aug 04 19:53:20.807576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.807153 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" Aug 04 19:53:20.807576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.807199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" event={"ID":"16415033-f7ab-483c-997b-91ff1bdd64a7","Type":"ContainerDied","Data":"d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e"} Aug 04 19:53:20.807576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.807245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds" event={"ID":"16415033-f7ab-483c-997b-91ff1bdd64a7","Type":"ContainerDied","Data":"e19b70f1a3d3501e4148917c67bf7eb8d54de90581842d7f1f5095eba5346e94"} Aug 04 19:53:20.807576 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.807269 2579 scope.go:117] "RemoveContainer" containerID="d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e" Aug 04 19:53:20.817180 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.817157 2579 scope.go:117] "RemoveContainer" containerID="d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e" Aug 04 19:53:20.817512 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:53:20.817488 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e\": container with ID starting with d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e not found: ID does not exist" containerID="d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e" Aug 04 19:53:20.817589 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.817521 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e"} err="failed to get container status \"d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e\": rpc error: code = NotFound desc = could not find container \"d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e\": container with ID starting with d1379a0843ba840c25275b466e31030e4a3a2bb9a9698d9a585e67fdb168aa1e not found: ID does not exist" Aug 04 19:53:20.828962 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.828933 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds"] Aug 04 19:53:20.834976 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:20.834948 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fb61cc13/e2e-lifecycle-model-fb61cc13-kserve-f5f6d4499-jxrds"] Aug 04 19:53:21.610888 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:21.610852 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16415033-f7ab-483c-997b-91ff1bdd64a7" path="/var/lib/kubelet/pods/16415033-f7ab-483c-997b-91ff1bdd64a7/volumes" Aug 04 19:53:47.746588 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:47.746550 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l"] Aug 04 19:53:47.747183 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:47.746860 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" containerID="cri-o://80e9f715742e77593875a86052390e4326371e35108b4edead13acd231060a18" gracePeriod=30 Aug 04 19:53:51.310424 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:51.310379 2579 patch_prober.go:28] interesting pod/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:53:51.310824 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:51.310445 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:53:52.971524 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:52.971486 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerID="80e9f715742e77593875a86052390e4326371e35108b4edead13acd231060a18" exitCode=0 Aug 04 19:53:52.972023 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:52.971554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" event={"ID":"0ef9a20f-efe5-4ffc-95f2-4df025516802","Type":"ContainerDied","Data":"80e9f715742e77593875a86052390e4326371e35108b4edead13acd231060a18"} Aug 04 19:53:52.972023 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:52.971625 2579 scope.go:117] "RemoveContainer" containerID="bff54a6221d9aba8f61cd3e5e67ffca93ef482865868a0d184bb0b1eb0b693e9" Aug 04 19:53:53.003830 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.003801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:53:53.150228 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-socket\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-token\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvc94\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-kube-api-access-qvc94\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9a20f-efe5-4ffc-95f2-4df025516802-istiod-ca-cert\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-podinfo\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150398 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-certs\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150434 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150422 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-credential-socket\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150442 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-data\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150500 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-envoy\") pod \"0ef9a20f-efe5-4ffc-95f2-4df025516802\" (UID: \"0ef9a20f-efe5-4ffc-95f2-4df025516802\") " Aug 04 19:53:53.150749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150577 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:53.150749 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150734 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef9a20f-efe5-4ffc-95f2-4df025516802-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:53:53.150950 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150763 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:53.150950 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150862 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.150950 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150917 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9a20f-efe5-4ffc-95f2-4df025516802-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.150950 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150932 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.151090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150944 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:53.151090 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.150956 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:53.152775 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.152742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:53:53.152907 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.152783 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:53:53.152907 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.152862 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:53:53.153069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.153049 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-kube-api-access-qvc94" (OuterVolumeSpecName: "kube-api-access-qvc94") pod "0ef9a20f-efe5-4ffc-95f2-4df025516802" (UID: "0ef9a20f-efe5-4ffc-95f2-4df025516802"). InnerVolumeSpecName "kube-api-access-qvc94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:53:53.252447 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.252412 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.252447 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.252439 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.252447 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.252449 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvc94\" (UniqueName: \"kubernetes.io/projected/0ef9a20f-efe5-4ffc-95f2-4df025516802-kube-api-access-qvc94\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.252682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.252459 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9a20f-efe5-4ffc-95f2-4df025516802-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.252682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.252470 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.252682 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.252478 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9a20f-efe5-4ffc-95f2-4df025516802-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:53:53.977602 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.977503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" event={"ID":"0ef9a20f-efe5-4ffc-95f2-4df025516802","Type":"ContainerDied","Data":"bd1b365d792076cb99b18262d21190922cf4de89c57e08360dbb1413af39343d"} Aug 04 19:53:53.977602 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.977528 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l" Aug 04 19:53:53.977602 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.977563 2579 scope.go:117] "RemoveContainer" containerID="80e9f715742e77593875a86052390e4326371e35108b4edead13acd231060a18" Aug 04 19:53:53.995518 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:53.995486 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l"] Aug 04 19:53:54.004075 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:54.001700 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fb61cc13-openshift-default-5487fb648d-b797l"] Aug 04 19:53:55.610813 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:53:55.610775 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" path="/var/lib/kubelet/pods/0ef9a20f-efe5-4ffc-95f2-4df025516802/volumes" Aug 04 19:54:53.256281 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.256235 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv"] Aug 04 19:54:53.257115 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257089 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16415033-f7ab-483c-997b-91ff1bdd64a7" containerName="main" Aug 04 19:54:53.257235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257118 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16415033-f7ab-483c-997b-91ff1bdd64a7" containerName="main" Aug 04 19:54:53.257235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257139 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" Aug 04 19:54:53.257235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257149 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" Aug 04 19:54:53.257235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257162 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" Aug 04 19:54:53.257235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257178 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" Aug 04 19:54:53.257516 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257370 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" Aug 04 19:54:53.257516 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257389 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef9a20f-efe5-4ffc-95f2-4df025516802" containerName="istio-proxy" Aug 04 19:54:53.257516 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.257399 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16415033-f7ab-483c-997b-91ff1bdd64a7" containerName="main" Aug 04 19:54:53.261623 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.261597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.264636 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.264607 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-48bed7-openshift-default-dockercfg-b84st\"" Aug 04 19:54:53.266236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.266208 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv"] Aug 04 19:54:53.313846 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.313811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-socket\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314047 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.313882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413842d6-675c-43b7-b658-ef10742639c5-istiod-ca-cert\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314047 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.313912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-data\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314047 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.313949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413842d6-675c-43b7-b658-ef10742639c5-istio-podinfo\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314047 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.314027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-istio-token\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.314102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9896\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-kube-api-access-m9896\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.314139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-credential-socket\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.314154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-envoy\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.314223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.314172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-certs\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415473 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413842d6-675c-43b7-b658-ef10742639c5-istiod-ca-cert\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-data\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413842d6-675c-43b7-b658-ef10742639c5-istio-podinfo\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-istio-token\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9896\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-kube-api-access-m9896\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-credential-socket\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.415680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-envoy\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416021 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-certs\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416021 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.415755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-socket\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.416036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-data\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416146 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.416078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-socket\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416378 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.416351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-credential-socket\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.416608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-certs\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.416742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.416696 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413842d6-675c-43b7-b658-ef10742639c5-istiod-ca-cert\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.418680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.418656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-envoy\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.419014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.418989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413842d6-675c-43b7-b658-ef10742639c5-istio-podinfo\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.424343 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.424296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-istio-token\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.424803 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.424775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9896\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-kube-api-access-m9896\") pod \"e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.574297 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.574246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:53.723078 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.723040 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv"] Aug 04 19:54:53.725468 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:54:53.725440 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413842d6_675c_43b7_b658_ef10742639c5.slice/crio-1efc39c7d37dfd515423907c4564aa60ad6673b091eba440ad802e19bee8e45c WatchSource:0}: Error finding container 1efc39c7d37dfd515423907c4564aa60ad6673b091eba440ad802e19bee8e45c: Status 404 returned error can't find the container with id 1efc39c7d37dfd515423907c4564aa60ad6673b091eba440ad802e19bee8e45c Aug 04 19:54:53.727596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.727564 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:54:53.727716 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.727641 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:54:53.727716 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:53.727686 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:54:54.269067 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:54.269030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" event={"ID":"413842d6-675c-43b7-b658-ef10742639c5","Type":"ContainerStarted","Data":"7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee"} Aug 04 19:54:54.269554 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:54.269091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" event={"ID":"413842d6-675c-43b7-b658-ef10742639c5","Type":"ContainerStarted","Data":"1efc39c7d37dfd515423907c4564aa60ad6673b091eba440ad802e19bee8e45c"} Aug 04 19:54:54.289420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:54.289323 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podStartSLOduration=1.289305825 podStartE2EDuration="1.289305825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:54:54.288109534 +0000 UTC m=+2371.228906281" watchObservedRunningTime="2026-08-04 19:54:54.289305825 +0000 UTC m=+2371.230102570" Aug 04 19:54:54.575235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:54.575193 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:54.580711 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:54.580681 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:55.274344 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:55.274294 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:54:55.275395 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:54:55.275375 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:55:10.270690 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:55:10.270654 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413842d6_675c_43b7_b658_ef10742639c5.slice/crio-conmon-7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee.scope\": RecentStats: unable to find data in memory cache]" Aug 04 19:55:10.275621 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.275589 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:10.275750 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.275653 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:10.350654 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.350620 2579 generic.go:356] "Generic (PLEG): container finished" podID="413842d6-675c-43b7-b658-ef10742639c5" containerID="7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee" exitCode=0 Aug 04 19:55:10.350838 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.350689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" event={"ID":"413842d6-675c-43b7-b658-ef10742639c5","Type":"ContainerDied","Data":"7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee"} Aug 04 19:55:10.351104 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.351090 2579 scope.go:117] "RemoveContainer" containerID="7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee" Aug 04 19:55:10.352097 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.352073 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:10.352154 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.352135 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:10.352194 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:10.352167 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:11.359526 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:11.359492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" event={"ID":"413842d6-675c-43b7-b658-ef10742639c5","Type":"ContainerStarted","Data":"69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5"} Aug 04 19:55:11.575032 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:11.574989 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:55:11.575236 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:11.575210 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:11.575290 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:11.575271 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:12.575315 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:12.575278 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:12.575847 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:12.575347 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:13.574891 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:13.574856 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:13.575081 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:13.574913 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:14.575061 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:14.575022 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:14.575497 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:14.575077 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:15.575277 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:15.575239 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:15.575716 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:15.575300 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:16.575639 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:16.575602 2579 patch_prober.go:28] interesting pod/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Aug 04 19:55:16.576036 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:16.575657 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Aug 04 19:55:17.579216 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:17.579179 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:55:17.579675 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:17.579585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:55:17.580430 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:17.580408 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:55:21.860072 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.860031 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96"] Aug 04 19:55:21.864142 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.864115 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.867584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.867556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ddb738-openshift-default-dockercfg-4k9m8\"" Aug 04 19:55:21.876299 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.876269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96"] Aug 04 19:55:21.899712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-certs\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.899712 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-envoy\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.899965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kzl\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-kube-api-access-k7kzl\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.899965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-istio-token\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.899965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-socket\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.899965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a336051d-cfe4-45a9-9995-521f5fab7928-istio-podinfo\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.899965 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-credential-socket\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.900158 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.899982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-data\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:21.900158 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:21.900009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a336051d-cfe4-45a9-9995-521f5fab7928-istiod-ca-cert\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.000660 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.000616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a336051d-cfe4-45a9-9995-521f5fab7928-istiod-ca-cert\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.000867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.000684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-certs\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.000867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.000720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-envoy\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.000867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.000751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kzl\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-kube-api-access-k7kzl\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.000867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.000818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-istio-token\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.000867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.000852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-socket\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001117 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a336051d-cfe4-45a9-9995-521f5fab7928-istio-podinfo\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001117 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-credential-socket\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-data\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-certs\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001221 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001205 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-socket\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001483 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001457 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-data\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-credential-socket\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.001685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.001670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a336051d-cfe4-45a9-9995-521f5fab7928-istiod-ca-cert\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.003809 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.003787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-envoy\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.003971 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.003957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a336051d-cfe4-45a9-9995-521f5fab7928-istio-podinfo\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.009292 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.009269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-istio-token\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.009584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.009562 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kzl\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-kube-api-access-k7kzl\") pod \"e2e-inf-b-ddb738-openshift-default-545c695956-k7j96\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.176820 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.176726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:22.335024 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.330006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96"] Aug 04 19:55:22.338756 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.338720 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:22.338844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.338792 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:22.338844 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.338820 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:22.416121 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.416077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" event={"ID":"a336051d-cfe4-45a9-9995-521f5fab7928","Type":"ContainerStarted","Data":"bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2"} Aug 04 19:55:22.416357 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.416150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" event={"ID":"a336051d-cfe4-45a9-9995-521f5fab7928","Type":"ContainerStarted","Data":"20ab5454d949fc89f65dbe575ebaa10faf42096ac1ff35c79d089f9c543d4446"} Aug 04 19:55:22.437772 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:22.437657 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podStartSLOduration=1.437639498 podStartE2EDuration="1.437639498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:55:22.435364867 +0000 UTC m=+2399.376161614" watchObservedRunningTime="2026-08-04 19:55:22.437639498 +0000 UTC m=+2399.378436245" Aug 04 19:55:23.177417 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:23.177370 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:23.182448 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:23.182406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:23.421208 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:23.421174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:23.422297 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:23.422272 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:23.732271 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:23.732207 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:55:23.734400 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:23.734373 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 19:55:38.422551 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.422514 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 04 19:55:38.423001 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.422582 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 04 19:55:38.502686 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.502655 2579 generic.go:356] "Generic (PLEG): container finished" podID="a336051d-cfe4-45a9-9995-521f5fab7928" containerID="bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2" exitCode=0 Aug 04 19:55:38.502871 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.502713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" event={"ID":"a336051d-cfe4-45a9-9995-521f5fab7928","Type":"ContainerDied","Data":"bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2"} Aug 04 19:55:38.503110 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.503097 2579 scope.go:117] "RemoveContainer" containerID="bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2" Aug 04 19:55:38.504144 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.504102 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:38.504268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.504213 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:38.504350 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:38.504277 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:55:39.510188 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:39.510150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" event={"ID":"a336051d-cfe4-45a9-9995-521f5fab7928","Type":"ContainerStarted","Data":"0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde"} Aug 04 19:55:40.177797 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:40.177753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:40.178026 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:40.178000 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 04 19:55:40.178093 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:40.178071 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 04 19:55:41.178027 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:41.177989 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 04 19:55:41.178551 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:41.178055 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 04 19:55:42.178199 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:42.178155 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 04 19:55:42.178639 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:42.178217 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 04 19:55:43.178129 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:43.178094 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 04 19:55:43.178308 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:43.178152 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 04 19:55:44.177643 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:44.177597 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Aug 04 19:55:44.177840 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:44.177668 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Aug 04 19:55:45.181516 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:45.181478 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:45.182113 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:45.181861 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:55:45.182550 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:55:45.182520 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:56:26.585753 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.585715 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p"] Aug 04 19:56:26.589309 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.589285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.592546 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.592515 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ddb738\"/\"test-model-ddb738-kserve-self-signed-certs\"" Aug 04 19:56:26.592546 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.592540 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ddb738\"/\"openshift-service-ca.crt\"" Aug 04 19:56:26.592827 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.592548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ddb738\"/\"default-dockercfg-nzc92\"" Aug 04 19:56:26.592827 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.592599 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ddb738\"/\"kube-root-ca.crt\"" Aug 04 19:56:26.601598 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.601565 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p"] Aug 04 19:56:26.727477 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.727427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-dshm\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.727477 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.727483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-model-cache\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.727703 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.727548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-home\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.727703 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.727603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tmp-dir\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.727703 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.727678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj57c\" (UniqueName: \"kubernetes.io/projected/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-kube-api-access-fj57c\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.727812 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.727765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tls-certs\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.828652 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.828598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-home\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.828652 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.828652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tmp-dir\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.828925 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.828723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj57c\" (UniqueName: \"kubernetes.io/projected/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-kube-api-access-fj57c\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.828925 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.828789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tls-certs\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.828925 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.828846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-dshm\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.828925 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.828872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-model-cache\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.829256 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.829066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-home\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.829256 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.829099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tmp-dir\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.829256 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.829227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-model-cache\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.831264 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.831239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-dshm\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.831520 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.831502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tls-certs\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.839645 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.839561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj57c\" (UniqueName: \"kubernetes.io/projected/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-kube-api-access-fj57c\") pod \"test-model-ddb738-kserve-659d4bf4f-95w8p\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:26.901511 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:26.901467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:27.043253 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:27.043224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p"] Aug 04 19:56:27.045917 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:56:27.045885 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fc53a2d_544f_4309_b6c5_9517ac3b7b4b.slice/crio-5de003698417a5c1a73b52516265320a82b57fe4b5ce3023751235d7f5acc50f WatchSource:0}: Error finding container 5de003698417a5c1a73b52516265320a82b57fe4b5ce3023751235d7f5acc50f: Status 404 returned error can't find the container with id 5de003698417a5c1a73b52516265320a82b57fe4b5ce3023751235d7f5acc50f Aug 04 19:56:27.753788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:27.753748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" event={"ID":"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b","Type":"ContainerStarted","Data":"7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50"} Aug 04 19:56:27.753788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:27.753789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" event={"ID":"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b","Type":"ContainerStarted","Data":"5de003698417a5c1a73b52516265320a82b57fe4b5ce3023751235d7f5acc50f"} Aug 04 19:56:27.774042 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:27.773983 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" podStartSLOduration=1.773962183 podStartE2EDuration="1.773962183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:56:27.771769112 +0000 UTC m=+2464.712565859" watchObservedRunningTime="2026-08-04 19:56:27.773962183 +0000 UTC m=+2464.714758928" Aug 04 19:56:36.902306 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:36.902259 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:36.902829 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:36.902374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:36.915080 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:36.915042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:56:37.814921 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:56:37.814886 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:57:10.042850 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:10.042807 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p"] Aug 04 19:57:25.062596 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.062481 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" podUID="2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" containerName="main" containerID="cri-o://7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50" gracePeriod=45 Aug 04 19:57:25.320614 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.320537 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:57:25.505618 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505574 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-dshm\") pod \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " Aug 04 19:57:25.505810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505629 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-home\") pod \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " Aug 04 19:57:25.505810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj57c\" (UniqueName: \"kubernetes.io/projected/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-kube-api-access-fj57c\") pod \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " Aug 04 19:57:25.505810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tmp-dir\") pod \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " Aug 04 19:57:25.505810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505782 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tls-certs\") pod \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " Aug 04 19:57:25.505810 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505811 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-model-cache\") pod \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\" (UID: \"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b\") " Aug 04 19:57:25.506080 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.505946 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-home" (OuterVolumeSpecName: "home") pod "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" (UID: "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:25.506132 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.506078 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" (UID: "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:25.506189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.506164 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" (UID: "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:25.506242 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.506192 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:25.506242 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.506212 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:25.508248 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.508212 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-kube-api-access-fj57c" (OuterVolumeSpecName: "kube-api-access-fj57c") pod "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" (UID: "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b"). InnerVolumeSpecName "kube-api-access-fj57c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:25.508396 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.508247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" (UID: "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 19:57:25.508452 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.508421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-dshm" (OuterVolumeSpecName: "dshm") pod "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" (UID: "2fc53a2d-544f-4309-b6c5-9517ac3b7b4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:25.607431 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.607339 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:25.607431 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.607366 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj57c\" (UniqueName: \"kubernetes.io/projected/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-kube-api-access-fj57c\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:25.607431 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.607378 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:25.607431 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:25.607388 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:26.048771 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.048737 2579 generic.go:356] "Generic (PLEG): container finished" podID="2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" containerID="7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50" exitCode=0 Aug 04 19:57:26.048955 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.048787 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" Aug 04 19:57:26.048955 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.048824 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" event={"ID":"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b","Type":"ContainerDied","Data":"7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50"} Aug 04 19:57:26.048955 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.048876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p" event={"ID":"2fc53a2d-544f-4309-b6c5-9517ac3b7b4b","Type":"ContainerDied","Data":"5de003698417a5c1a73b52516265320a82b57fe4b5ce3023751235d7f5acc50f"} Aug 04 19:57:26.048955 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.048899 2579 scope.go:117] "RemoveContainer" containerID="7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50" Aug 04 19:57:26.058573 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.058549 2579 scope.go:117] "RemoveContainer" containerID="7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50" Aug 04 19:57:26.058890 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:57:26.058870 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50\": container with ID starting with 7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50 not found: ID does not exist" containerID="7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50" Aug 04 19:57:26.058942 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.058902 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50"} err="failed to get container status \"7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50\": rpc error: code = NotFound desc = could not find container \"7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50\": container with ID starting with 7316baeae06ef3efe0a9a5bc2bf0b2c89238f3c5b6ce7a81fe2d781f5e121f50 not found: ID does not exist" Aug 04 19:57:26.064842 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.064804 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p"] Aug 04 19:57:26.068446 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:26.068419 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ddb738/test-model-ddb738-kserve-659d4bf4f-95w8p"] Aug 04 19:57:27.318557 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:27.318517 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv"] Aug 04 19:57:27.318954 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:27.318771 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" containerID="cri-o://69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5" gracePeriod=30 Aug 04 19:57:27.611035 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:27.610932 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" path="/var/lib/kubelet/pods/2fc53a2d-544f-4309-b6c5-9517ac3b7b4b/volumes" Aug 04 19:57:32.580894 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.580868 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:57:32.674168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674054 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-credential-socket\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674115 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9896\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-kube-api-access-m9896\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674137 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-envoy\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674168 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-istio-token\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674548 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674214 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413842d6-675c-43b7-b658-ef10742639c5-istio-podinfo\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674548 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-socket\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674548 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-data\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674548 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-certs\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674548 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674377 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413842d6-675c-43b7-b658-ef10742639c5-istiod-ca-cert\") pod \"413842d6-675c-43b7-b658-ef10742639c5\" (UID: \"413842d6-675c-43b7-b658-ef10742639c5\") " Aug 04 19:57:32.674548 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674408 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:32.674854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674629 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:32.674854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:32.674854 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674746 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413842d6-675c-43b7-b658-ef10742639c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:32.675014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674877 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.675014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674895 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.675014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674908 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413842d6-675c-43b7-b658-ef10742639c5-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.675014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674921 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.675014 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.674932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:32.676535 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.676509 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:32.676638 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.676616 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/413842d6-675c-43b7-b658-ef10742639c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:57:32.676939 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.676913 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-kube-api-access-m9896" (OuterVolumeSpecName: "kube-api-access-m9896") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "kube-api-access-m9896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:32.676939 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.676921 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "413842d6-675c-43b7-b658-ef10742639c5" (UID: "413842d6-675c-43b7-b658-ef10742639c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:32.775964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.775908 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9896\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-kube-api-access-m9896\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.775964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.775955 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.775964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.775965 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413842d6-675c-43b7-b658-ef10742639c5-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.775964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.775973 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413842d6-675c-43b7-b658-ef10742639c5-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:32.775964 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:32.775981 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413842d6-675c-43b7-b658-ef10742639c5-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:33.086479 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.086443 2579 generic.go:356] "Generic (PLEG): container finished" podID="413842d6-675c-43b7-b658-ef10742639c5" containerID="69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5" exitCode=0 Aug 04 19:57:33.086661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.086493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" Aug 04 19:57:33.086661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.086521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" event={"ID":"413842d6-675c-43b7-b658-ef10742639c5","Type":"ContainerDied","Data":"69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5"} Aug 04 19:57:33.086661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.086574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv" event={"ID":"413842d6-675c-43b7-b658-ef10742639c5","Type":"ContainerDied","Data":"1efc39c7d37dfd515423907c4564aa60ad6673b091eba440ad802e19bee8e45c"} Aug 04 19:57:33.086661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.086596 2579 scope.go:117] "RemoveContainer" containerID="69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5" Aug 04 19:57:33.096629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.096610 2579 scope.go:117] "RemoveContainer" containerID="7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee" Aug 04 19:57:33.105710 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.105679 2579 scope.go:117] "RemoveContainer" containerID="69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5" Aug 04 19:57:33.106531 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:57:33.106208 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5\": container with ID starting with 69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5 not found: ID does not exist" containerID="69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5" Aug 04 19:57:33.106531 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.106254 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5"} err="failed to get container status \"69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5\": rpc error: code = NotFound desc = could not find container \"69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5\": container with ID starting with 69c35e97af32f7a3e5cff4623a07fb155064f717848fd2a157fdeecced9ffee5 not found: ID does not exist" Aug 04 19:57:33.106531 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.106278 2579 scope.go:117] "RemoveContainer" containerID="7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee" Aug 04 19:57:33.106733 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:57:33.106698 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee\": container with ID starting with 7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee not found: ID does not exist" containerID="7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee" Aug 04 19:57:33.106778 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.106729 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee"} err="failed to get container status \"7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee\": rpc error: code = NotFound desc = could not find container \"7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee\": container with ID starting with 7bded2e4c2379cb2b620fb225b4a45232a73c72832864ee0fe520bb4fc50d5ee not found: ID does not exist" Aug 04 19:57:33.108739 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.108719 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv"] Aug 04 19:57:33.111704 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.111675 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-48bed7-openshift-default-5bddfdffcd-6njzv"] Aug 04 19:57:33.610463 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:33.610429 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413842d6-675c-43b7-b658-ef10742639c5" path="/var/lib/kubelet/pods/413842d6-675c-43b7-b658-ef10742639c5/volumes" Aug 04 19:57:44.709443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:44.709402 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96"] Aug 04 19:57:44.709983 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:44.709646 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" containerID="cri-o://0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde" gracePeriod=30 Aug 04 19:57:45.184855 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.184822 2579 patch_prober.go:28] interesting pod/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 19:57:45.185068 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.184878 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 19:57:45.600041 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600002 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8"] Aug 04 19:57:45.600616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600599 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" Aug 04 19:57:45.600681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600620 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" Aug 04 19:57:45.600681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600653 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" containerName="main" Aug 04 19:57:45.600681 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600662 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" containerName="main" Aug 04 19:57:45.600788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600681 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" Aug 04 19:57:45.600788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600691 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" Aug 04 19:57:45.600852 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600794 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" Aug 04 19:57:45.600852 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc53a2d-544f-4309-b6c5-9517ac3b7b4b" containerName="main" Aug 04 19:57:45.600852 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.600818 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="413842d6-675c-43b7-b658-ef10742639c5" containerName="istio-proxy" Aug 04 19:57:45.605766 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.605736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.608043 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.608014 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4feefd-openshift-default-dockercfg-xswv6\"" Aug 04 19:57:45.617316 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.617287 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8"] Aug 04 19:57:45.704888 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.704832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-data\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.704899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-credential-socket\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.704931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-podinfo\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.704964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-envoy\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705084 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.705017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-certs\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705257 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.705092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istiod-ca-cert\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705257 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.705123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-socket\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705257 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.705185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-token\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.705400 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.705236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwzr\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-kube-api-access-brwzr\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.806504 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-data\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-credential-socket\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-podinfo\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-envoy\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-certs\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istiod-ca-cert\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-socket\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-token\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwzr\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-kube-api-access-brwzr\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-data\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807006 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.806996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-credential-socket\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807547 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.807048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-certs\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807547 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.807372 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-socket\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.807547 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.807420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istiod-ca-cert\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.809107 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.809083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-envoy\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.809237 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.809219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-podinfo\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.814594 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.814566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwzr\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-kube-api-access-brwzr\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.814703 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.814620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-token\") pod \"e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:45.918067 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:45.917964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:46.070916 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.070881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8"] Aug 04 19:57:46.073320 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:57:46.073282 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56fccdf_9e09_48bd_ab9d_da0ef83ee683.slice/crio-dec6b7a8ee9297f1ca396ce45d0e0841792458745f73f23efee5013787468995 WatchSource:0}: Error finding container dec6b7a8ee9297f1ca396ce45d0e0841792458745f73f23efee5013787468995: Status 404 returned error can't find the container with id dec6b7a8ee9297f1ca396ce45d0e0841792458745f73f23efee5013787468995 Aug 04 19:57:46.075423 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.075360 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:57:46.075537 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.075455 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:57:46.075537 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.075484 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:57:46.154629 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.154584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" event={"ID":"b56fccdf-9e09-48bd-ab9d-da0ef83ee683","Type":"ContainerStarted","Data":"71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c"} Aug 04 19:57:46.154794 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.154665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" event={"ID":"b56fccdf-9e09-48bd-ab9d-da0ef83ee683","Type":"ContainerStarted","Data":"dec6b7a8ee9297f1ca396ce45d0e0841792458745f73f23efee5013787468995"} Aug 04 19:57:46.174550 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.174412 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podStartSLOduration=1.174388148 podStartE2EDuration="1.174388148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:57:46.172186896 +0000 UTC m=+2543.112983644" watchObservedRunningTime="2026-08-04 19:57:46.174388148 +0000 UTC m=+2543.115184895" Aug 04 19:57:46.918552 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:46.918510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:47.923493 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:47.923459 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:48.164692 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:48.164650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:48.165680 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:48.165660 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:57:49.978980 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:49.978906 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:57:50.049897 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.049866 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-certs\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.049897 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.049903 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-socket\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.049959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a336051d-cfe4-45a9-9995-521f5fab7928-istiod-ca-cert\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a336051d-cfe4-45a9-9995-521f5fab7928-istio-podinfo\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-credential-socket\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-envoy\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050111 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7kzl\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-kube-api-access-k7kzl\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-istio-token\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050177 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-data\") pod \"a336051d-cfe4-45a9-9995-521f5fab7928\" (UID: \"a336051d-cfe4-45a9-9995-521f5fab7928\") " Aug 04 19:57:50.050604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050218 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:50.050604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050261 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:50.050604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050364 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:50.050604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050553 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.050604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050576 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.050604 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050591 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.050916 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050583 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-data" (OuterVolumeSpecName: "istio-data") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:50.050916 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.050736 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a336051d-cfe4-45a9-9995-521f5fab7928-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:57:50.052443 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.052408 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a336051d-cfe4-45a9-9995-521f5fab7928-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:57:50.052554 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.052441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:57:50.052554 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.052498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-kube-api-access-k7kzl" (OuterVolumeSpecName: "kube-api-access-k7kzl") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "kube-api-access-k7kzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:50.052730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.052708 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-istio-token" (OuterVolumeSpecName: "istio-token") pod "a336051d-cfe4-45a9-9995-521f5fab7928" (UID: "a336051d-cfe4-45a9-9995-521f5fab7928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:57:50.151845 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.151796 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a336051d-cfe4-45a9-9995-521f5fab7928-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.151845 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.151832 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.151845 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.151848 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kzl\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-kube-api-access-k7kzl\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.152116 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.151861 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a336051d-cfe4-45a9-9995-521f5fab7928-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.152116 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.151876 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a336051d-cfe4-45a9-9995-521f5fab7928-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.152116 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.151887 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a336051d-cfe4-45a9-9995-521f5fab7928-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:57:50.177261 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.177225 2579 generic.go:356] "Generic (PLEG): container finished" podID="a336051d-cfe4-45a9-9995-521f5fab7928" containerID="0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde" exitCode=0 Aug 04 19:57:50.177458 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.177274 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" Aug 04 19:57:50.177458 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.177305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" event={"ID":"a336051d-cfe4-45a9-9995-521f5fab7928","Type":"ContainerDied","Data":"0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde"} Aug 04 19:57:50.177458 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.177398 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96" event={"ID":"a336051d-cfe4-45a9-9995-521f5fab7928","Type":"ContainerDied","Data":"20ab5454d949fc89f65dbe575ebaa10faf42096ac1ff35c79d089f9c543d4446"} Aug 04 19:57:50.177458 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.177432 2579 scope.go:117] "RemoveContainer" containerID="0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde" Aug 04 19:57:50.188914 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.188880 2579 scope.go:117] "RemoveContainer" containerID="bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2" Aug 04 19:57:50.198254 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.198208 2579 scope.go:117] "RemoveContainer" containerID="0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde" Aug 04 19:57:50.198685 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:57:50.198654 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde\": container with ID starting with 0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde not found: ID does not exist" containerID="0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde" Aug 04 19:57:50.198807 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.198695 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde"} err="failed to get container status \"0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde\": rpc error: code = NotFound desc = could not find container \"0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde\": container with ID starting with 0c4305ce0da4e4ec8eed71570eaf5a51f23a49a69c62854707481a4ac6d5dcde not found: ID does not exist" Aug 04 19:57:50.198807 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.198716 2579 scope.go:117] "RemoveContainer" containerID="bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2" Aug 04 19:57:50.199012 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:57:50.198986 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2\": container with ID starting with bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2 not found: ID does not exist" containerID="bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2" Aug 04 19:57:50.199078 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.199022 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2"} err="failed to get container status \"bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2\": rpc error: code = NotFound desc = could not find container \"bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2\": container with ID starting with bde72cbbaf37183e8fc90cec8b3221b404d26f8c50ccf857c77bf5302677fef2 not found: ID does not exist" Aug 04 19:57:50.200371 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.200347 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96"] Aug 04 19:57:50.205341 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:50.205301 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ddb738-openshift-default-545c695956-k7j96"] Aug 04 19:57:51.611064 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:57:51.611025 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" path="/var/lib/kubelet/pods/a336051d-cfe4-45a9-9995-521f5fab7928/volumes" Aug 04 19:58:02.239697 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:02.239660 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerID="71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c" exitCode=0 Aug 04 19:58:02.240136 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:02.239731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" event={"ID":"b56fccdf-9e09-48bd-ab9d-da0ef83ee683","Type":"ContainerDied","Data":"71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c"} Aug 04 19:58:02.240189 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:02.240135 2579 scope.go:117] "RemoveContainer" containerID="71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c" Aug 04 19:58:02.241169 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:02.241144 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:02.241223 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:02.241211 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:02.241264 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:02.241242 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:03.165055 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:03.165010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:58:03.251268 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:03.251223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" event={"ID":"b56fccdf-9e09-48bd-ab9d-da0ef83ee683","Type":"ContainerStarted","Data":"d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1"} Aug 04 19:58:03.918558 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:03.918518 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:58:03.919702 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:03.919671 2579 patch_prober.go:28] interesting pod/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Aug 04 19:58:03.919829 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:03.919725 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Aug 04 19:58:04.919040 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:04.919002 2579 patch_prober.go:28] interesting pod/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Aug 04 19:58:04.919499 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:04.919070 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Aug 04 19:58:05.918841 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:05.918800 2579 patch_prober.go:28] interesting pod/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Aug 04 19:58:05.919038 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:05.918875 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Aug 04 19:58:06.918788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:06.918748 2579 patch_prober.go:28] interesting pod/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Aug 04 19:58:06.919203 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:06.918818 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Aug 04 19:58:07.918627 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:07.918587 2579 patch_prober.go:28] interesting pod/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Aug 04 19:58:07.918819 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:07.918648 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Aug 04 19:58:08.923392 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:08.923354 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:58:08.923782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:08.923663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:58:08.924345 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:08.924311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:58:13.852757 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.852722 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s"] Aug 04 19:58:13.853136 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.853094 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" Aug 04 19:58:13.853136 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.853107 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" Aug 04 19:58:13.853217 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.853173 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" Aug 04 19:58:13.853217 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.853181 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" Aug 04 19:58:13.856607 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.856579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.859172 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.859143 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2e00c3-openshift-default-dockercfg-hrs56\"" Aug 04 19:58:13.868785 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.868753 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s"] Aug 04 19:58:13.883748 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.883700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-certs\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.883748 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.883764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-istio-token\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.883873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a25b8-1513-4576-b7a1-72fe374f059e-istio-podinfo\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.883912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-envoy\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.883934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr2cg\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-kube-api-access-mr2cg\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.883998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a25b8-1513-4576-b7a1-72fe374f059e-istiod-ca-cert\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.884029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-data\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.884057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-credential-socket\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.884318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.884108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-socket\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985158 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-credential-socket\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985158 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-socket\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-certs\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-istio-token\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a25b8-1513-4576-b7a1-72fe374f059e-istio-podinfo\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-envoy\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr2cg\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-kube-api-access-mr2cg\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a25b8-1513-4576-b7a1-72fe374f059e-istiod-ca-cert\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985469 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-data\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.985824 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.985651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-credential-socket\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.986059 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.986036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-certs\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.986141 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.986068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-data\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.986183 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.986157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-socket\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.986465 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.986441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a25b8-1513-4576-b7a1-72fe374f059e-istiod-ca-cert\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.988253 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.988227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a25b8-1513-4576-b7a1-72fe374f059e-istio-podinfo\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.988437 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.988416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-envoy\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.993583 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.993556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-istio-token\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:13.993813 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:13.993791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr2cg\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-kube-api-access-mr2cg\") pod \"e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:14.170259 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:14.170155 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:14.323281 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:14.323243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s"] Aug 04 19:58:14.327163 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:58:14.327131 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335a25b8_1513_4576_b7a1_72fe374f059e.slice/crio-5da52dbfe5638ce97ffed045a0f99a35f1e7a01a9e59e976d7522c6a68fe2170 WatchSource:0}: Error finding container 5da52dbfe5638ce97ffed045a0f99a35f1e7a01a9e59e976d7522c6a68fe2170: Status 404 returned error can't find the container with id 5da52dbfe5638ce97ffed045a0f99a35f1e7a01a9e59e976d7522c6a68fe2170 Aug 04 19:58:14.329577 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:14.329542 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:14.329676 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:14.329613 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:14.329676 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:14.329648 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:15.317761 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:15.317725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" event={"ID":"335a25b8-1513-4576-b7a1-72fe374f059e","Type":"ContainerStarted","Data":"967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b"} Aug 04 19:58:15.318174 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:15.317779 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" event={"ID":"335a25b8-1513-4576-b7a1-72fe374f059e","Type":"ContainerStarted","Data":"5da52dbfe5638ce97ffed045a0f99a35f1e7a01a9e59e976d7522c6a68fe2170"} Aug 04 19:58:15.338437 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:15.338372 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podStartSLOduration=2.338350639 podStartE2EDuration="2.338350639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:58:15.337502582 +0000 UTC m=+2572.278299345" watchObservedRunningTime="2026-08-04 19:58:15.338350639 +0000 UTC m=+2572.279147386" Aug 04 19:58:16.171150 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:16.171106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:16.176311 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:16.176288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:16.323319 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:16.323286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:16.324433 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:16.324414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:31.325069 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.325032 2579 patch_prober.go:28] interesting pod/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Aug 04 19:58:31.325531 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.325099 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Aug 04 19:58:31.402367 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.402313 2579 generic.go:356] "Generic (PLEG): container finished" podID="335a25b8-1513-4576-b7a1-72fe374f059e" containerID="967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b" exitCode=0 Aug 04 19:58:31.402537 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.402385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" event={"ID":"335a25b8-1513-4576-b7a1-72fe374f059e","Type":"ContainerDied","Data":"967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b"} Aug 04 19:58:31.402841 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.402820 2579 scope.go:117] "RemoveContainer" containerID="967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b" Aug 04 19:58:31.404661 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.404622 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:31.404973 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.404949 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:31.405210 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:31.405180 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:58:32.409003 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:32.408970 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" event={"ID":"335a25b8-1513-4576-b7a1-72fe374f059e","Type":"ContainerStarted","Data":"3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c"} Aug 04 19:58:33.170685 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:33.170642 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:33.170886 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:33.170861 2579 patch_prober.go:28] interesting pod/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Aug 04 19:58:33.170946 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:33.170925 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Aug 04 19:58:34.170829 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:34.170790 2579 patch_prober.go:28] interesting pod/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Aug 04 19:58:34.171260 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:34.170862 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Aug 04 19:58:35.171456 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:35.171420 2579 patch_prober.go:28] interesting pod/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Aug 04 19:58:35.171894 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:35.171480 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Aug 04 19:58:36.170677 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:36.170637 2579 patch_prober.go:28] interesting pod/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Aug 04 19:58:36.170861 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:36.170702 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Aug 04 19:58:37.171178 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:37.171140 2579 patch_prober.go:28] interesting pod/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Aug 04 19:58:37.171632 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:37.171202 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Aug 04 19:58:38.175571 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:38.175537 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:38.176026 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:38.175896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:58:38.176788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:58:38.176769 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:59:01.504528 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:01.504436 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8"] Aug 04 19:59:01.505164 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:01.504666 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" containerID="cri-o://d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1" gracePeriod=30 Aug 04 19:59:06.765796 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.765764 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:59:06.911664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911620 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-credential-socket\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911664 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911670 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-podinfo\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911943 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911691 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-data\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911943 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911738 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-token\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911943 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911786 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-socket\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911943 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911819 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istiod-ca-cert\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911943 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911873 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brwzr\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-kube-api-access-brwzr\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.911943 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911933 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-envoy\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.912263 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-certs\") pod \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\" (UID: \"b56fccdf-9e09-48bd-ab9d-da0ef83ee683\") " Aug 04 19:59:06.912263 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.911963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:06.912263 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.912136 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:06.912460 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.912380 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:06.912460 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.912414 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:06.912460 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.912379 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:06.912460 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.912428 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:59:06.912670 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.912507 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-data" (OuterVolumeSpecName: "istio-data") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:06.914207 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.914169 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:59:06.914207 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.914192 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-token" (OuterVolumeSpecName: "istio-token") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:06.914410 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.914201 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-kube-api-access-brwzr" (OuterVolumeSpecName: "kube-api-access-brwzr") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "kube-api-access-brwzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:06.914410 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:06.914246 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b56fccdf-9e09-48bd-ab9d-da0ef83ee683" (UID: "b56fccdf-9e09-48bd-ab9d-da0ef83ee683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:07.013690 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013653 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.013690 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013684 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.013690 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013693 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwzr\" (UniqueName: \"kubernetes.io/projected/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-kube-api-access-brwzr\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.013936 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013705 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.013936 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013715 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.013936 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013724 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.013936 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.013731 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56fccdf-9e09-48bd-ab9d-da0ef83ee683-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:07.590270 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.590233 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerID="d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1" exitCode=0 Aug 04 19:59:07.590478 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.590277 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" Aug 04 19:59:07.590478 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.590312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" event={"ID":"b56fccdf-9e09-48bd-ab9d-da0ef83ee683","Type":"ContainerDied","Data":"d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1"} Aug 04 19:59:07.590478 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.590382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8" event={"ID":"b56fccdf-9e09-48bd-ab9d-da0ef83ee683","Type":"ContainerDied","Data":"dec6b7a8ee9297f1ca396ce45d0e0841792458745f73f23efee5013787468995"} Aug 04 19:59:07.590478 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.590407 2579 scope.go:117] "RemoveContainer" containerID="d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1" Aug 04 19:59:07.600376 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.600349 2579 scope.go:117] "RemoveContainer" containerID="71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c" Aug 04 19:59:07.609379 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.609360 2579 scope.go:117] "RemoveContainer" containerID="d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1" Aug 04 19:59:07.609665 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:59:07.609646 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1\": container with ID starting with d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1 not found: ID does not exist" containerID="d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1" Aug 04 19:59:07.609720 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.609675 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1"} err="failed to get container status \"d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1\": rpc error: code = NotFound desc = could not find container \"d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1\": container with ID starting with d90c4dc030ae30dd29172c4ac80df61bb4fabda3119b10d2b0ebe1e18d9607e1 not found: ID does not exist" Aug 04 19:59:07.609720 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.609698 2579 scope.go:117] "RemoveContainer" containerID="71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c" Aug 04 19:59:07.609972 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:59:07.609954 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c\": container with ID starting with 71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c not found: ID does not exist" containerID="71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c" Aug 04 19:59:07.610041 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.609977 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c"} err="failed to get container status \"71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c\": rpc error: code = NotFound desc = could not find container \"71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c\": container with ID starting with 71603671dd1743ee7712c9153b439285126697964c243f4c1b2548346115a90c not found: ID does not exist" Aug 04 19:59:07.618989 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.618952 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8"] Aug 04 19:59:07.624400 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:07.624369 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4feefd-openshift-default-7565fb5596-wfpv8"] Aug 04 19:59:09.610168 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:09.610133 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" path="/var/lib/kubelet/pods/b56fccdf-9e09-48bd-ab9d-da0ef83ee683/volumes" Aug 04 19:59:24.245379 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:24.245260 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s"] Aug 04 19:59:24.245852 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:24.245684 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" containerID="cri-o://3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c" gracePeriod=30 Aug 04 19:59:25.215285 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215250 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5"] Aug 04 19:59:25.215695 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215681 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" Aug 04 19:59:25.215742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215697 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" Aug 04 19:59:25.215742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215707 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" Aug 04 19:59:25.215742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215713 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" Aug 04 19:59:25.215742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215736 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" Aug 04 19:59:25.215742 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215742 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a336051d-cfe4-45a9-9995-521f5fab7928" containerName="istio-proxy" Aug 04 19:59:25.215898 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215804 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" Aug 04 19:59:25.215898 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.215811 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56fccdf-9e09-48bd-ab9d-da0ef83ee683" containerName="istio-proxy" Aug 04 19:59:25.218946 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.218923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.221476 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.221454 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-03ae76-openshift-default-dockercfg-jqbv6\"" Aug 04 19:59:25.243895 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.243850 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5"] Aug 04 19:59:25.390424 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-token\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390424 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-envoy\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-certs\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-credential-socket\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-data\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-socket\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-podinfo\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istiod-ca-cert\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.390988 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.390707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqgq\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-kube-api-access-gtqgq\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491444 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-token\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491598 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-envoy\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491598 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-certs\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491598 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-credential-socket\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491733 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-data\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491788 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-socket\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491873 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-podinfo\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491954 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istiod-ca-cert\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.491954 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-credential-socket\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.492073 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-certs\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.492073 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.491965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqgq\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-kube-api-access-gtqgq\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.492073 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.492046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-socket\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.492241 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.492118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-data\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.492553 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.492532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istiod-ca-cert\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.493963 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.493944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-envoy\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.494289 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.494269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-podinfo\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.501054 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.501022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqgq\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-kube-api-access-gtqgq\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.502034 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.502010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-token\") pod \"e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.530131 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.530090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:25.686018 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.685981 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5"] Aug 04 19:59:25.688641 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:59:25.688603 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9e0f86_4b81_49a3_9e58_dae1b91eb65d.slice/crio-b11cd45ec002f929f93692369a2746cb583a9c3e2ac3327767b6dac54974f9c3 WatchSource:0}: Error finding container b11cd45ec002f929f93692369a2746cb583a9c3e2ac3327767b6dac54974f9c3: Status 404 returned error can't find the container with id b11cd45ec002f929f93692369a2746cb583a9c3e2ac3327767b6dac54974f9c3 Aug 04 19:59:25.690895 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.690858 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:25.690992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.690926 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:25.690992 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:25.690958 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:26.684873 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:26.684828 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" event={"ID":"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d","Type":"ContainerStarted","Data":"779f2a21c687de3c9935628d074866d00b2478f96c59a7b22fdf22eccdcf4f50"} Aug 04 19:59:26.685276 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:26.684897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" event={"ID":"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d","Type":"ContainerStarted","Data":"b11cd45ec002f929f93692369a2746cb583a9c3e2ac3327767b6dac54974f9c3"} Aug 04 19:59:26.704776 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:26.704718 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podStartSLOduration=1.7046987310000001 podStartE2EDuration="1.704698731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:59:26.703149231 +0000 UTC m=+2643.643945988" watchObservedRunningTime="2026-08-04 19:59:26.704698731 +0000 UTC m=+2643.645495477" Aug 04 19:59:27.530999 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:27.530950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:27.536784 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:27.536754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:27.689603 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:27.689571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:27.690630 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:27.690611 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:29.501893 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.501865 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:59:29.535571 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a25b8-1513-4576-b7a1-72fe374f059e-istiod-ca-cert\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535608 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-istio-token\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535644 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-certs\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535670 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a25b8-1513-4576-b7a1-72fe374f059e-istio-podinfo\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535730 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-socket\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535754 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-data\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-envoy\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535826 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr2cg\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-kube-api-access-mr2cg\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535866 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-credential-socket\") pod \"335a25b8-1513-4576-b7a1-72fe374f059e\" (UID: \"335a25b8-1513-4576-b7a1-72fe374f059e\") " Aug 04 19:59:29.535991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335a25b8-1513-4576-b7a1-72fe374f059e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 19:59:29.535991 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.535962 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:29.536292 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536070 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-data" (OuterVolumeSpecName: "istio-data") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:29.536292 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:29.536420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536293 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:29.536420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536356 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.536420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536378 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.536420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536392 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a25b8-1513-4576-b7a1-72fe374f059e-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.536420 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.536405 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.538170 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.538146 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/335a25b8-1513-4576-b7a1-72fe374f059e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 19:59:29.538577 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.538559 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-istio-token" (OuterVolumeSpecName: "istio-token") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:29.538662 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.538586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 19:59:29.538744 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.538728 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-kube-api-access-mr2cg" (OuterVolumeSpecName: "kube-api-access-mr2cg") pod "335a25b8-1513-4576-b7a1-72fe374f059e" (UID: "335a25b8-1513-4576-b7a1-72fe374f059e"). InnerVolumeSpecName "kube-api-access-mr2cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 19:59:29.637249 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.637215 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.637249 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.637246 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr2cg\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-kube-api-access-mr2cg\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.637249 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.637256 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a25b8-1513-4576-b7a1-72fe374f059e-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.637584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.637265 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a25b8-1513-4576-b7a1-72fe374f059e-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.637584 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.637273 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a25b8-1513-4576-b7a1-72fe374f059e-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 19:59:29.701185 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.701151 2579 generic.go:356] "Generic (PLEG): container finished" podID="335a25b8-1513-4576-b7a1-72fe374f059e" containerID="3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c" exitCode=0 Aug 04 19:59:29.701390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.701204 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" Aug 04 19:59:29.701390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.701231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" event={"ID":"335a25b8-1513-4576-b7a1-72fe374f059e","Type":"ContainerDied","Data":"3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c"} Aug 04 19:59:29.701390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.701285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s" event={"ID":"335a25b8-1513-4576-b7a1-72fe374f059e","Type":"ContainerDied","Data":"5da52dbfe5638ce97ffed045a0f99a35f1e7a01a9e59e976d7522c6a68fe2170"} Aug 04 19:59:29.701390 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.701308 2579 scope.go:117] "RemoveContainer" containerID="3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c" Aug 04 19:59:29.710743 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.710716 2579 scope.go:117] "RemoveContainer" containerID="967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b" Aug 04 19:59:29.720035 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.720008 2579 scope.go:117] "RemoveContainer" containerID="3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c" Aug 04 19:59:29.720418 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.720396 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s"] Aug 04 19:59:29.720506 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:59:29.720392 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c\": container with ID starting with 3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c not found: ID does not exist" containerID="3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c" Aug 04 19:59:29.720506 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.720456 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c"} err="failed to get container status \"3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c\": rpc error: code = NotFound desc = could not find container \"3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c\": container with ID starting with 3636909f237938770c3d1a533a1cc0100de88ef1477541c45a47e24e8a04023c not found: ID does not exist" Aug 04 19:59:29.720506 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.720481 2579 scope.go:117] "RemoveContainer" containerID="967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b" Aug 04 19:59:29.720799 ip-10-0-143-62 kubenswrapper[2579]: E0804 19:59:29.720763 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b\": container with ID starting with 967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b not found: ID does not exist" containerID="967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b" Aug 04 19:59:29.720887 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.720794 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b"} err="failed to get container status \"967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b\": rpc error: code = NotFound desc = could not find container \"967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b\": container with ID starting with 967301fda98704ccc0ff441118e963fcf6eb8653120136b9da14707443112b9b not found: ID does not exist" Aug 04 19:59:29.726220 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:29.726185 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2e00c3-openshift-default-5dc9d7b496-g8g9s"] Aug 04 19:59:31.610914 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:31.610874 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" path="/var/lib/kubelet/pods/335a25b8-1513-4576-b7a1-72fe374f059e/volumes" Aug 04 19:59:42.690951 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.690911 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 04 19:59:42.691406 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.690980 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 04 19:59:42.774694 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.774658 2579 generic.go:356] "Generic (PLEG): container finished" podID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerID="779f2a21c687de3c9935628d074866d00b2478f96c59a7b22fdf22eccdcf4f50" exitCode=0 Aug 04 19:59:42.774867 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.774729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" event={"ID":"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d","Type":"ContainerDied","Data":"779f2a21c687de3c9935628d074866d00b2478f96c59a7b22fdf22eccdcf4f50"} Aug 04 19:59:42.775235 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.775221 2579 scope.go:117] "RemoveContainer" containerID="779f2a21c687de3c9935628d074866d00b2478f96c59a7b22fdf22eccdcf4f50" Aug 04 19:59:42.776214 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.776189 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:42.776282 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.776268 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:42.776323 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:42.776309 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:43.781318 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:43.781268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" event={"ID":"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d","Type":"ContainerStarted","Data":"49038576108ccd8afa63ed38e75d7154702cb9493562ae3c87ee97e429363294"} Aug 04 19:59:44.530672 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:44.530632 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:44.530954 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:44.530928 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 04 19:59:44.531042 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:44.530999 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 04 19:59:45.531251 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:45.531215 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 04 19:59:45.531770 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:45.531272 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 04 19:59:46.530953 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:46.530917 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 04 19:59:46.531141 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:46.530989 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 04 19:59:47.531432 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:47.531390 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 04 19:59:47.532056 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:47.532021 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 04 19:59:48.531616 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:48.531577 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Aug 04 19:59:48.532206 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:48.532170 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Aug 04 19:59:49.535211 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:49.535171 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:49.535622 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:49.535572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:49.536343 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:49.536311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 19:59:54.331046 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.331009 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st"] Aug 04 19:59:54.331490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.331443 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" Aug 04 19:59:54.331490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.331463 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" Aug 04 19:59:54.331490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.331473 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" Aug 04 19:59:54.331490 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.331478 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" Aug 04 19:59:54.331648 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.331566 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" Aug 04 19:59:54.334782 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.334761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.338019 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.337995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-15a874-openshift-default-dockercfg-lqzlj\"" Aug 04 19:59:54.353792 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.353757 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st"] Aug 04 19:59:54.481805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m769x\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-kube-api-access-m769x\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.481805 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-envoy\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-credential-socket\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istiod-ca-cert\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-data\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-podinfo\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482044 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.481960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-token\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482205 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.482040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-certs\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.482205 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.482077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-socket\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.582793 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-podinfo\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.582793 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-token\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-certs\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-socket\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m769x\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-kube-api-access-m769x\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-envoy\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-credential-socket\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istiod-ca-cert\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.582969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-data\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583419 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.583283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-certs\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583419 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.583368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-socket\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583492 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.583455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-credential-socket\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583744 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.583719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-data\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.583812 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.583796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istiod-ca-cert\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.585431 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.585409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-envoy\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.585546 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.585507 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-podinfo\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.593600 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.593567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-token\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.593874 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.593854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m769x\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-kube-api-access-m769x\") pod \"e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.648220 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.648180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:54.809716 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.809688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st"] Aug 04 19:59:54.812193 ip-10-0-143-62 kubenswrapper[2579]: W0804 19:59:54.812158 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468d6e7b_a855_4c97_9f46_ec8b36d237b8.slice/crio-2fad920d0323901417a85d87cb2ca801fdad88dafc6886b1f17d54bca64b9b3d WatchSource:0}: Error finding container 2fad920d0323901417a85d87cb2ca801fdad88dafc6886b1f17d54bca64b9b3d: Status 404 returned error can't find the container with id 2fad920d0323901417a85d87cb2ca801fdad88dafc6886b1f17d54bca64b9b3d Aug 04 19:59:54.815180 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.815141 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:54.815265 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.815227 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:54.815304 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.815264 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 19:59:54.842151 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:54.842111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" event={"ID":"468d6e7b-a855-4c97-9f46-ec8b36d237b8","Type":"ContainerStarted","Data":"2fad920d0323901417a85d87cb2ca801fdad88dafc6886b1f17d54bca64b9b3d"} Aug 04 19:59:55.849163 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:55.849121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" event={"ID":"468d6e7b-a855-4c97-9f46-ec8b36d237b8","Type":"ContainerStarted","Data":"b0ee25bf8d662cd72d52a92cf2bd3b4fa8257f1e51e37769f77c459398bb1d4c"} Aug 04 19:59:55.868369 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:55.868289 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podStartSLOduration=1.868269974 podStartE2EDuration="1.868269974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:59:55.867225726 +0000 UTC m=+2672.808022473" watchObservedRunningTime="2026-08-04 19:59:55.868269974 +0000 UTC m=+2672.809066720" Aug 04 19:59:56.648930 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:56.648893 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:56.654737 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:56.654707 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:56.853907 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:56.853869 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 19:59:56.855029 ip-10-0-143-62 kubenswrapper[2579]: I0804 19:59:56.855008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:00:11.856080 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.856039 2579 patch_prober.go:28] interesting pod/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Aug 04 20:00:11.856568 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.856101 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Aug 04 20:00:11.930032 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.929998 2579 generic.go:356] "Generic (PLEG): container finished" podID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerID="b0ee25bf8d662cd72d52a92cf2bd3b4fa8257f1e51e37769f77c459398bb1d4c" exitCode=0 Aug 04 20:00:11.930195 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.930069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" event={"ID":"468d6e7b-a855-4c97-9f46-ec8b36d237b8","Type":"ContainerDied","Data":"b0ee25bf8d662cd72d52a92cf2bd3b4fa8257f1e51e37769f77c459398bb1d4c"} Aug 04 20:00:11.930520 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.930505 2579 scope.go:117] "RemoveContainer" containerID="b0ee25bf8d662cd72d52a92cf2bd3b4fa8257f1e51e37769f77c459398bb1d4c" Aug 04 20:00:11.931648 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.931622 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:00:11.931727 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.931693 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:00:11.931769 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:11.931726 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 20:00:12.940973 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:12.940930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" event={"ID":"468d6e7b-a855-4c97-9f46-ec8b36d237b8","Type":"ContainerStarted","Data":"b8bab173df208a6c26d64ea73865df51194bacf3c50b8b5ad9302c2cad550063"} Aug 04 20:00:13.648977 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:13.648931 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:00:13.649215 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:13.649180 2579 patch_prober.go:28] interesting pod/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Aug 04 20:00:13.649287 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:13.649232 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Aug 04 20:00:14.649275 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:14.649227 2579 patch_prober.go:28] interesting pod/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Aug 04 20:00:14.649976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:14.649922 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Aug 04 20:00:15.649176 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:15.649081 2579 patch_prober.go:28] interesting pod/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Aug 04 20:00:15.649514 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:15.649474 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Aug 04 20:00:16.648616 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:16.648574 2579 patch_prober.go:28] interesting pod/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Aug 04 20:00:16.648946 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:16.648915 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Aug 04 20:00:17.649622 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:17.649580 2579 patch_prober.go:28] interesting pod/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Aug 04 20:00:17.650269 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:17.650226 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Aug 04 20:00:18.652909 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:18.652871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:00:18.653297 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:18.653135 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:00:18.653789 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:18.653771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:00:23.767472 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:23.767444 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:00:23.770899 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:23.770871 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:00:33.571987 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.571941 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5"] Aug 04 20:00:33.572525 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.572444 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a25b8-1513-4576-b7a1-72fe374f059e" containerName="istio-proxy" Aug 04 20:00:33.574644 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.574614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.577858 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.577833 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"kube-root-ca.crt\"" Aug 04 20:00:33.578023 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.577874 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"default-dockercfg-z7f97\"" Aug 04 20:00:33.578023 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.577833 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"openshift-service-ca.crt\"" Aug 04 20:00:33.578023 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.577939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"auth-test-model-03ae76-kserve-self-signed-certs\"" Aug 04 20:00:33.586197 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.586167 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5"] Aug 04 20:00:33.672625 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.672583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-home\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.672839 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.672641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rh2w\" (UniqueName: \"kubernetes.io/projected/47b7ff1c-bd50-447d-add5-aeee1afb0260-kube-api-access-9rh2w\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.672839 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.672739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-dshm\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.672839 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.672813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-model-cache\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.673015 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.672849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b7ff1c-bd50-447d-add5-aeee1afb0260-tls-certs\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.673015 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.672888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-tmp-dir\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.773990 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.773949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-model-cache\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774140 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.773999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b7ff1c-bd50-447d-add5-aeee1afb0260-tls-certs\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774140 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-tmp-dir\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774140 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-home\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774140 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rh2w\" (UniqueName: \"kubernetes.io/projected/47b7ff1c-bd50-447d-add5-aeee1afb0260-kube-api-access-9rh2w\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774323 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-dshm\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774486 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-model-cache\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774563 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774503 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-tmp-dir\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.774636 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.774578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-home\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.776596 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.776574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-dshm\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.776852 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.776835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b7ff1c-bd50-447d-add5-aeee1afb0260-tls-certs\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.782251 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.782227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rh2w\" (UniqueName: \"kubernetes.io/projected/47b7ff1c-bd50-447d-add5-aeee1afb0260-kube-api-access-9rh2w\") pod \"auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:33.886243 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:33.886142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:34.027435 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:34.027403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5"] Aug 04 20:00:34.030308 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:00:34.030279 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47b7ff1c_bd50_447d_add5_aeee1afb0260.slice/crio-b6db8735b5e57993d2197414f7469f0594b9cf4240f9128e7212bbd802a7eb99 WatchSource:0}: Error finding container b6db8735b5e57993d2197414f7469f0594b9cf4240f9128e7212bbd802a7eb99: Status 404 returned error can't find the container with id b6db8735b5e57993d2197414f7469f0594b9cf4240f9128e7212bbd802a7eb99 Aug 04 20:00:34.054437 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:34.054406 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" event={"ID":"47b7ff1c-bd50-447d-add5-aeee1afb0260","Type":"ContainerStarted","Data":"b6db8735b5e57993d2197414f7469f0594b9cf4240f9128e7212bbd802a7eb99"} Aug 04 20:00:35.060841 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:35.060800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" event={"ID":"47b7ff1c-bd50-447d-add5-aeee1afb0260","Type":"ContainerStarted","Data":"95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903"} Aug 04 20:00:35.079515 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:35.079458 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" podStartSLOduration=2.079437197 podStartE2EDuration="2.079437197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:00:35.077842201 +0000 UTC m=+2712.018638947" watchObservedRunningTime="2026-08-04 20:00:35.079437197 +0000 UTC m=+2712.020233943" Aug 04 20:00:43.887254 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:43.887202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:43.887254 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:43.887265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:43.900248 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:43.900202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:00:44.120454 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:00:44.120417 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:05:23.797191 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:05:23.797162 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:05:23.801186 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:05:23.801152 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:05:46.582642 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:05:46.582604 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5"] Aug 04 20:06:01.602275 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.602210 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" podUID="47b7ff1c-bd50-447d-add5-aeee1afb0260" containerName="main" containerID="cri-o://95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903" gracePeriod=45 Aug 04 20:06:01.859978 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.859900 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:06:01.899965 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.899927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-model-cache\") pod \"47b7ff1c-bd50-447d-add5-aeee1afb0260\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " Aug 04 20:06:01.900173 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.899986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-tmp-dir\") pod \"47b7ff1c-bd50-447d-add5-aeee1afb0260\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " Aug 04 20:06:01.900173 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rh2w\" (UniqueName: \"kubernetes.io/projected/47b7ff1c-bd50-447d-add5-aeee1afb0260-kube-api-access-9rh2w\") pod \"47b7ff1c-bd50-447d-add5-aeee1afb0260\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " Aug 04 20:06:01.900173 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900115 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-home\") pod \"47b7ff1c-bd50-447d-add5-aeee1afb0260\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " Aug 04 20:06:01.900173 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-dshm\") pod \"47b7ff1c-bd50-447d-add5-aeee1afb0260\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " Aug 04 20:06:01.900452 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900196 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b7ff1c-bd50-447d-add5-aeee1afb0260-tls-certs\") pod \"47b7ff1c-bd50-447d-add5-aeee1afb0260\" (UID: \"47b7ff1c-bd50-447d-add5-aeee1afb0260\") " Aug 04 20:06:01.900452 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900246 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-model-cache" (OuterVolumeSpecName: "model-cache") pod "47b7ff1c-bd50-447d-add5-aeee1afb0260" (UID: "47b7ff1c-bd50-447d-add5-aeee1afb0260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:01.900452 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47b7ff1c-bd50-447d-add5-aeee1afb0260" (UID: "47b7ff1c-bd50-447d-add5-aeee1afb0260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:01.900452 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900405 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-home" (OuterVolumeSpecName: "home") pod "47b7ff1c-bd50-447d-add5-aeee1afb0260" (UID: "47b7ff1c-bd50-447d-add5-aeee1afb0260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:01.900667 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900597 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:06:01.900667 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900616 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:06:01.900667 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.900628 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:06:01.902418 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.902388 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b7ff1c-bd50-447d-add5-aeee1afb0260-kube-api-access-9rh2w" (OuterVolumeSpecName: "kube-api-access-9rh2w") pod "47b7ff1c-bd50-447d-add5-aeee1afb0260" (UID: "47b7ff1c-bd50-447d-add5-aeee1afb0260"). InnerVolumeSpecName "kube-api-access-9rh2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:06:01.902706 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.902689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b7ff1c-bd50-447d-add5-aeee1afb0260-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47b7ff1c-bd50-447d-add5-aeee1afb0260" (UID: "47b7ff1c-bd50-447d-add5-aeee1afb0260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:06:01.902783 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:01.902705 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-dshm" (OuterVolumeSpecName: "dshm") pod "47b7ff1c-bd50-447d-add5-aeee1afb0260" (UID: "47b7ff1c-bd50-447d-add5-aeee1afb0260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:06:02.001587 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.001547 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rh2w\" (UniqueName: \"kubernetes.io/projected/47b7ff1c-bd50-447d-add5-aeee1afb0260-kube-api-access-9rh2w\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.001587 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.001580 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47b7ff1c-bd50-447d-add5-aeee1afb0260-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.001587 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.001589 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47b7ff1c-bd50-447d-add5-aeee1afb0260-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:06:02.684790 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.684755 2579 generic.go:356] "Generic (PLEG): container finished" podID="47b7ff1c-bd50-447d-add5-aeee1afb0260" containerID="95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903" exitCode=0 Aug 04 20:06:02.684790 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.684795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" event={"ID":"47b7ff1c-bd50-447d-add5-aeee1afb0260","Type":"ContainerDied","Data":"95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903"} Aug 04 20:06:02.685248 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.684825 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" event={"ID":"47b7ff1c-bd50-447d-add5-aeee1afb0260","Type":"ContainerDied","Data":"b6db8735b5e57993d2197414f7469f0594b9cf4240f9128e7212bbd802a7eb99"} Aug 04 20:06:02.685248 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.684843 2579 scope.go:117] "RemoveContainer" containerID="95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903" Aug 04 20:06:02.685248 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.684796 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5" Aug 04 20:06:02.697661 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.697633 2579 scope.go:117] "RemoveContainer" containerID="95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903" Aug 04 20:06:02.698110 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:06:02.698084 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903\": container with ID starting with 95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903 not found: ID does not exist" containerID="95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903" Aug 04 20:06:02.698236 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.698124 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903"} err="failed to get container status \"95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903\": rpc error: code = NotFound desc = could not find container \"95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903\": container with ID starting with 95792cf669523e2917f1b442e94c81715102427c0d263c3b07c8c0f4a47f9903 not found: ID does not exist" Aug 04 20:06:02.707849 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.707819 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5"] Aug 04 20:06:02.711458 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:02.711435 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/auth-test-model-03ae76-kserve-7bcfcd74c5-lv5h5"] Aug 04 20:06:03.615881 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:03.615846 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b7ff1c-bd50-447d-add5-aeee1afb0260" path="/var/lib/kubelet/pods/47b7ff1c-bd50-447d-add5-aeee1afb0260/volumes" Aug 04 20:06:04.907057 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.907021 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5"] Aug 04 20:06:04.907445 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.907413 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b7ff1c-bd50-447d-add5-aeee1afb0260" containerName="main" Aug 04 20:06:04.907445 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.907425 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b7ff1c-bd50-447d-add5-aeee1afb0260" containerName="main" Aug 04 20:06:04.907533 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.907480 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b7ff1c-bd50-447d-add5-aeee1afb0260" containerName="main" Aug 04 20:06:04.910453 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.910434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:04.913470 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.913444 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"sub-test-model-03ae76-kserve-self-signed-certs\"" Aug 04 20:06:04.913747 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.913724 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"default-dockercfg-z7f97\"" Aug 04 20:06:04.913876 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.913725 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"openshift-service-ca.crt\"" Aug 04 20:06:04.913876 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.913727 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-03ae76\"/\"kube-root-ca.crt\"" Aug 04 20:06:04.919524 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:04.919498 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5"] Aug 04 20:06:05.033493 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.033450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-home\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.033684 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.033522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xcc\" (UniqueName: \"kubernetes.io/projected/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-kube-api-access-v5xcc\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.033684 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.033557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-model-cache\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.033684 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.033578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tmp-dir\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.033684 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.033627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tls-certs\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.033843 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.033691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-dshm\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.134757 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.134715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-model-cache\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.134976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.134764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tmp-dir\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.134976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.134794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tls-certs\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.134976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.134837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-dshm\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.134976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.134896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-home\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.134976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.134953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xcc\" (UniqueName: \"kubernetes.io/projected/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-kube-api-access-v5xcc\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.135239 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.135199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tmp-dir\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.135239 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.135230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-model-cache\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.135407 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.135306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-home\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.137714 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.137689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-dshm\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.137972 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.137952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tls-certs\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.143300 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.143277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xcc\" (UniqueName: \"kubernetes.io/projected/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-kube-api-access-v5xcc\") pod \"sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.221465 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.221367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:05.358448 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.358419 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5"] Aug 04 20:06:05.360420 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:06:05.360390 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf52b3f9a_7fb1_4b4b_8349_30da0cb1dc4e.slice/crio-313ad4f79c3aec274a909f8604eef7ca89fcb707c2b086a721d0c746ba8a22ab WatchSource:0}: Error finding container 313ad4f79c3aec274a909f8604eef7ca89fcb707c2b086a721d0c746ba8a22ab: Status 404 returned error can't find the container with id 313ad4f79c3aec274a909f8604eef7ca89fcb707c2b086a721d0c746ba8a22ab Aug 04 20:06:05.701315 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.701275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" event={"ID":"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e","Type":"ContainerStarted","Data":"fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb"} Aug 04 20:06:05.701315 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.701323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" event={"ID":"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e","Type":"ContainerStarted","Data":"313ad4f79c3aec274a909f8604eef7ca89fcb707c2b086a721d0c746ba8a22ab"} Aug 04 20:06:05.722095 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:05.722037 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" podStartSLOduration=1.722021561 podStartE2EDuration="1.722021561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:06:05.719915919 +0000 UTC m=+3042.660712696" watchObservedRunningTime="2026-08-04 20:06:05.722021561 +0000 UTC m=+3042.662818307" Aug 04 20:06:15.222383 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:15.222317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:15.222811 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:15.222400 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:15.235082 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:15.235041 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:06:15.760180 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:06:15.760146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:07:53.561585 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:07:53.561483 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5"] Aug 04 20:08:08.580826 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.580758 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" podUID="f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" containerName="main" containerID="cri-o://fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb" gracePeriod=45 Aug 04 20:08:08.837803 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.837730 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:08:08.954046 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-dshm\") pod \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " Aug 04 20:08:08.954250 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tls-certs\") pod \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " Aug 04 20:08:08.954250 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954131 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5xcc\" (UniqueName: \"kubernetes.io/projected/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-kube-api-access-v5xcc\") pod \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " Aug 04 20:08:08.954250 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954151 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-home\") pod \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " Aug 04 20:08:08.954250 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tmp-dir\") pod \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " Aug 04 20:08:08.954250 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954230 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-model-cache\") pod \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\" (UID: \"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e\") " Aug 04 20:08:08.954576 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954473 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-home" (OuterVolumeSpecName: "home") pod "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" (UID: "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:08.954576 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954493 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" (UID: "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:08.954666 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.954598 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" (UID: "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:08.956403 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.956378 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" (UID: "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:08:08.956515 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.956376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-dshm" (OuterVolumeSpecName: "dshm") pod "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" (UID: "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:08:08.956515 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:08.956440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-kube-api-access-v5xcc" (OuterVolumeSpecName: "kube-api-access-v5xcc") pod "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" (UID: "f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e"). InnerVolumeSpecName "kube-api-access-v5xcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:08:09.055656 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.055611 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xcc\" (UniqueName: \"kubernetes.io/projected/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-kube-api-access-v5xcc\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.055656 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.055648 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.055656 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.055663 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.055977 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.055674 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.055977 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.055686 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.055977 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.055696 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:08:09.311796 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.311761 2579 generic.go:356] "Generic (PLEG): container finished" podID="f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" containerID="fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb" exitCode=0 Aug 04 20:08:09.311959 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.311799 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" Aug 04 20:08:09.311959 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.311808 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" event={"ID":"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e","Type":"ContainerDied","Data":"fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb"} Aug 04 20:08:09.311959 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.311845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5" event={"ID":"f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e","Type":"ContainerDied","Data":"313ad4f79c3aec274a909f8604eef7ca89fcb707c2b086a721d0c746ba8a22ab"} Aug 04 20:08:09.311959 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.311862 2579 scope.go:117] "RemoveContainer" containerID="fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb" Aug 04 20:08:09.321844 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.321824 2579 scope.go:117] "RemoveContainer" containerID="fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb" Aug 04 20:08:09.322171 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:08:09.322149 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb\": container with ID starting with fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb not found: ID does not exist" containerID="fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb" Aug 04 20:08:09.322237 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.322181 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb"} err="failed to get container status \"fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb\": rpc error: code = NotFound desc = could not find container \"fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb\": container with ID starting with fb60d9a8c8f6e1a1e19820a78eb2b35e25966969c96d14a15897bad9bbfb53fb not found: ID does not exist" Aug 04 20:08:09.333018 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.332991 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5"] Aug 04 20:08:09.337187 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.337163 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-03ae76/sub-test-model-03ae76-kserve-c6f9d8b67-x5mn5"] Aug 04 20:08:09.611303 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:09.611221 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" path="/var/lib/kubelet/pods/f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e/volumes" Aug 04 20:08:18.008433 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.008382 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2"] Aug 04 20:08:18.009096 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.009016 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" containerName="main" Aug 04 20:08:18.009096 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.009037 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" containerName="main" Aug 04 20:08:18.009218 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.009158 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f52b3f9a-7fb1-4b4b-8349-30da0cb1dc4e" containerName="main" Aug 04 20:08:18.014419 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.014298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.018057 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.017808 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"openshift-service-ca.crt\"" Aug 04 20:08:18.018057 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.017879 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"rate-test-model-15a874-kserve-self-signed-certs\"" Aug 04 20:08:18.018057 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.017926 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"kube-root-ca.crt\"" Aug 04 20:08:18.018057 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.017807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-15a874\"/\"default-dockercfg-8c6w4\"" Aug 04 20:08:18.020365 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.020312 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2"] Aug 04 20:08:18.143884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.143604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-tmp-dir\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.143884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.143655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5723188b-ea5a-43d3-a94c-204b94fc181c-tls-certs\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.143884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.143696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzck\" (UniqueName: \"kubernetes.io/projected/5723188b-ea5a-43d3-a94c-204b94fc181c-kube-api-access-9fzck\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.143884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.143745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-dshm\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.143884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.143790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-home\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.143884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.143898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-model-cache\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.244600 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.244555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-home\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.244797 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.244654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-model-cache\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.244797 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.244687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-tmp-dir\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.244797 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.244703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5723188b-ea5a-43d3-a94c-204b94fc181c-tls-certs\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.244797 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.244726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzck\" (UniqueName: \"kubernetes.io/projected/5723188b-ea5a-43d3-a94c-204b94fc181c-kube-api-access-9fzck\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.244797 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.244757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-dshm\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.245098 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.245069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-home\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.245171 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.245080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-model-cache\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.245171 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.245145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-tmp-dir\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.247097 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.247065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-dshm\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.247291 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.247276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5723188b-ea5a-43d3-a94c-204b94fc181c-tls-certs\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.253730 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.253700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzck\" (UniqueName: \"kubernetes.io/projected/5723188b-ea5a-43d3-a94c-204b94fc181c-kube-api-access-9fzck\") pod \"rate-test-model-15a874-kserve-7d4457656c-65bg2\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.326360 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.326286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:18.463972 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:18.463941 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2"] Aug 04 20:08:18.465778 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:08:18.465750 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5723188b_ea5a_43d3_a94c_204b94fc181c.slice/crio-059dde58076f0c179ab071e8756474436ba5becbd3dd5ed2d85f5187591e2c4c WatchSource:0}: Error finding container 059dde58076f0c179ab071e8756474436ba5becbd3dd5ed2d85f5187591e2c4c: Status 404 returned error can't find the container with id 059dde58076f0c179ab071e8756474436ba5becbd3dd5ed2d85f5187591e2c4c Aug 04 20:08:19.362136 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:19.362101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" event={"ID":"5723188b-ea5a-43d3-a94c-204b94fc181c","Type":"ContainerStarted","Data":"6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491"} Aug 04 20:08:19.362549 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:19.362145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" event={"ID":"5723188b-ea5a-43d3-a94c-204b94fc181c","Type":"ContainerStarted","Data":"059dde58076f0c179ab071e8756474436ba5becbd3dd5ed2d85f5187591e2c4c"} Aug 04 20:08:19.379816 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:19.379757 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" podStartSLOduration=2.379740575 podStartE2EDuration="2.379740575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:08:19.3786346 +0000 UTC m=+3176.319431344" watchObservedRunningTime="2026-08-04 20:08:19.379740575 +0000 UTC m=+3176.320537361" Aug 04 20:08:28.326992 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:28.326950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:28.327465 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:28.327009 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:28.339709 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:28.339682 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:08:28.416869 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:08:28.416837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:09:42.564714 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:42.564675 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2"] Aug 04 20:09:57.585370 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.585288 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" podUID="5723188b-ea5a-43d3-a94c-204b94fc181c" containerName="main" containerID="cri-o://6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491" gracePeriod=45 Aug 04 20:09:57.840367 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.840267 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:09:57.863174 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.863144 2579 generic.go:356] "Generic (PLEG): container finished" podID="5723188b-ea5a-43d3-a94c-204b94fc181c" containerID="6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491" exitCode=0 Aug 04 20:09:57.863369 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.863188 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" Aug 04 20:09:57.863369 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.863204 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" event={"ID":"5723188b-ea5a-43d3-a94c-204b94fc181c","Type":"ContainerDied","Data":"6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491"} Aug 04 20:09:57.863369 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.863238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2" event={"ID":"5723188b-ea5a-43d3-a94c-204b94fc181c","Type":"ContainerDied","Data":"059dde58076f0c179ab071e8756474436ba5becbd3dd5ed2d85f5187591e2c4c"} Aug 04 20:09:57.863369 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.863258 2579 scope.go:117] "RemoveContainer" containerID="6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491" Aug 04 20:09:57.872318 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.872297 2579 scope.go:117] "RemoveContainer" containerID="6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491" Aug 04 20:09:57.872643 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:09:57.872614 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491\": container with ID starting with 6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491 not found: ID does not exist" containerID="6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491" Aug 04 20:09:57.872716 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.872657 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491"} err="failed to get container status \"6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491\": rpc error: code = NotFound desc = could not find container \"6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491\": container with ID starting with 6146c2dce67bd7f8c92a935b1d0119eb52cc6e34fda26c69f4f0e2015b6f5491 not found: ID does not exist" Aug 04 20:09:57.942843 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.942810 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-home\") pod \"5723188b-ea5a-43d3-a94c-204b94fc181c\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " Aug 04 20:09:57.943032 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.942858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-tmp-dir\") pod \"5723188b-ea5a-43d3-a94c-204b94fc181c\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " Aug 04 20:09:57.943032 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.942907 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-model-cache\") pod \"5723188b-ea5a-43d3-a94c-204b94fc181c\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " Aug 04 20:09:57.943032 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.942931 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fzck\" (UniqueName: \"kubernetes.io/projected/5723188b-ea5a-43d3-a94c-204b94fc181c-kube-api-access-9fzck\") pod \"5723188b-ea5a-43d3-a94c-204b94fc181c\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " Aug 04 20:09:57.943032 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.942974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-dshm\") pod \"5723188b-ea5a-43d3-a94c-204b94fc181c\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " Aug 04 20:09:57.943032 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5723188b-ea5a-43d3-a94c-204b94fc181c-tls-certs\") pod \"5723188b-ea5a-43d3-a94c-204b94fc181c\" (UID: \"5723188b-ea5a-43d3-a94c-204b94fc181c\") " Aug 04 20:09:57.943300 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943100 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-home" (OuterVolumeSpecName: "home") pod "5723188b-ea5a-43d3-a94c-204b94fc181c" (UID: "5723188b-ea5a-43d3-a94c-204b94fc181c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.943300 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943186 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5723188b-ea5a-43d3-a94c-204b94fc181c" (UID: "5723188b-ea5a-43d3-a94c-204b94fc181c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.943300 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943201 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5723188b-ea5a-43d3-a94c-204b94fc181c" (UID: "5723188b-ea5a-43d3-a94c-204b94fc181c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.943499 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943449 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.943499 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943470 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.943499 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.943485 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:09:57.945281 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.945255 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-dshm" (OuterVolumeSpecName: "dshm") pod "5723188b-ea5a-43d3-a94c-204b94fc181c" (UID: "5723188b-ea5a-43d3-a94c-204b94fc181c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:09:57.945281 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.945263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5723188b-ea5a-43d3-a94c-204b94fc181c-kube-api-access-9fzck" (OuterVolumeSpecName: "kube-api-access-9fzck") pod "5723188b-ea5a-43d3-a94c-204b94fc181c" (UID: "5723188b-ea5a-43d3-a94c-204b94fc181c"). InnerVolumeSpecName "kube-api-access-9fzck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:09:57.945463 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:57.945284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5723188b-ea5a-43d3-a94c-204b94fc181c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5723188b-ea5a-43d3-a94c-204b94fc181c" (UID: "5723188b-ea5a-43d3-a94c-204b94fc181c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:09:58.044364 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:58.044294 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5723188b-ea5a-43d3-a94c-204b94fc181c-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:09:58.044364 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:58.044353 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5723188b-ea5a-43d3-a94c-204b94fc181c-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:09:58.044364 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:58.044366 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzck\" (UniqueName: \"kubernetes.io/projected/5723188b-ea5a-43d3-a94c-204b94fc181c-kube-api-access-9fzck\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:09:58.185764 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:58.185723 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2"] Aug 04 20:09:58.189214 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:58.189184 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-15a874/rate-test-model-15a874-kserve-7d4457656c-65bg2"] Aug 04 20:09:59.610824 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:09:59.610793 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5723188b-ea5a-43d3-a94c-204b94fc181c" path="/var/lib/kubelet/pods/5723188b-ea5a-43d3-a94c-204b94fc181c/volumes" Aug 04 20:10:23.827305 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:23.827170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:10:23.837506 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:23.837470 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:10:39.542678 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.541886 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw"] Aug 04 20:10:39.545289 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.543282 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5723188b-ea5a-43d3-a94c-204b94fc181c" containerName="main" Aug 04 20:10:39.545289 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.543310 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5723188b-ea5a-43d3-a94c-204b94fc181c" containerName="main" Aug 04 20:10:39.545289 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.543591 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5723188b-ea5a-43d3-a94c-204b94fc181c" containerName="main" Aug 04 20:10:39.547566 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.547541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.550599 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.550573 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9cd851-dockercfg-ffvrw\"" Aug 04 20:10:39.550866 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.550847 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9cd851\"" Aug 04 20:10:39.551442 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.551417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw"] Aug 04 20:10:39.646986 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.646941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7260a-a0d4-431d-a994-78e876e6dc1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.647182 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.647000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjft\" (UniqueName: \"kubernetes.io/projected/e0d7260a-a0d4-431d-a994-78e876e6dc1c-kube-api-access-rqjft\") pod \"payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.748486 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.748439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7260a-a0d4-431d-a994-78e876e6dc1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.748693 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.748509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjft\" (UniqueName: \"kubernetes.io/projected/e0d7260a-a0d4-431d-a994-78e876e6dc1c-kube-api-access-rqjft\") pod \"payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.749291 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.749261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7260a-a0d4-431d-a994-78e876e6dc1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.757507 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.757481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjft\" (UniqueName: \"kubernetes.io/projected/e0d7260a-a0d4-431d-a994-78e876e6dc1c-kube-api-access-rqjft\") pod \"payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:39.861904 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:39.861811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:40.010356 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:40.010244 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw"] Aug 04 20:10:40.013288 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:10:40.013258 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d7260a_a0d4_431d_a994_78e876e6dc1c.slice/crio-103411358eb1d9cd0af13b53eed43dcbb8283578340a124c648a487237134838 WatchSource:0}: Error finding container 103411358eb1d9cd0af13b53eed43dcbb8283578340a124c648a487237134838: Status 404 returned error can't find the container with id 103411358eb1d9cd0af13b53eed43dcbb8283578340a124c648a487237134838 Aug 04 20:10:40.015318 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:40.015298 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 20:10:40.070086 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:40.070050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" event={"ID":"e0d7260a-a0d4-431d-a994-78e876e6dc1c","Type":"ContainerStarted","Data":"103411358eb1d9cd0af13b53eed43dcbb8283578340a124c648a487237134838"} Aug 04 20:10:43.088164 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:43.088123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" event={"ID":"e0d7260a-a0d4-431d-a994-78e876e6dc1c","Type":"ContainerStarted","Data":"32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b"} Aug 04 20:10:43.088669 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:43.088187 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:10:43.103102 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:43.103050 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" podStartSLOduration=1.6532325879999998 podStartE2EDuration="4.103032716s" totalImagesPullingTime="2.449800128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:10:43.100731138 +0000 UTC m=+3320.041527883" watchObservedRunningTime="2026-08-04 20:10:43.103032716 +0000 UTC m=+3320.043829462" Aug 04 20:10:54.095574 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:10:54.095491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:11:09.777207 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:09.777176 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw_e0d7260a-a0d4-431d-a994-78e876e6dc1c/payload-processing/0.log" Aug 04 20:11:26.591462 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.591407 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb"] Aug 04 20:11:26.599896 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.599873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.603709 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.603677 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-41ab60\"/\"kube-root-ca.crt\"" Aug 04 20:11:26.604030 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.604009 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-41ab60\"/\"openshift-service-ca.crt\"" Aug 04 20:11:26.604126 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.604089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-41ab60\"/\"default-dockercfg-gtjnx\"" Aug 04 20:11:26.604126 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.604115 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-41ab60\"/\"ipp-route-41ab60-kserve-self-signed-certs\"" Aug 04 20:11:26.619951 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.619913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb"] Aug 04 20:11:26.692131 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.692090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-dshm\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.692320 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.692193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-tmp-dir\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.692320 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.692239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d86966-aef0-4b84-b131-c1613a54a2ba-tls-certs\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.692320 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.692271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-home\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.692320 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.692302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvj5\" (UniqueName: \"kubernetes.io/projected/34d86966-aef0-4b84-b131-c1613a54a2ba-kube-api-access-ksvj5\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.692515 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.692361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-model-cache\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793191 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-dshm\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793414 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-tmp-dir\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793414 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d86966-aef0-4b84-b131-c1613a54a2ba-tls-certs\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793562 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-home\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793562 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvj5\" (UniqueName: \"kubernetes.io/projected/34d86966-aef0-4b84-b131-c1613a54a2ba-kube-api-access-ksvj5\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793562 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-model-cache\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793789 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-tmp-dir\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793903 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-model-cache\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.793980 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.793934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-home\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.795746 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.795716 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-dshm\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.796010 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.795991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d86966-aef0-4b84-b131-c1613a54a2ba-tls-certs\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.802461 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.802434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvj5\" (UniqueName: \"kubernetes.io/projected/34d86966-aef0-4b84-b131-c1613a54a2ba-kube-api-access-ksvj5\") pod \"ipp-route-41ab60-kserve-d5b5c8cff-zcwwb\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:26.910353 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:26.910244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:27.050510 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:27.050484 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb"] Aug 04 20:11:27.052517 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:11:27.052485 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34d86966_aef0_4b84_b131_c1613a54a2ba.slice/crio-0e3b192c1d193f74865712c0ec501be652e2bb130c05cf2de9ec6f577e66384d WatchSource:0}: Error finding container 0e3b192c1d193f74865712c0ec501be652e2bb130c05cf2de9ec6f577e66384d: Status 404 returned error can't find the container with id 0e3b192c1d193f74865712c0ec501be652e2bb130c05cf2de9ec6f577e66384d Aug 04 20:11:27.304382 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:27.304341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" event={"ID":"34d86966-aef0-4b84-b131-c1613a54a2ba","Type":"ContainerStarted","Data":"3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249"} Aug 04 20:11:27.304568 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:27.304392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" event={"ID":"34d86966-aef0-4b84-b131-c1613a54a2ba","Type":"ContainerStarted","Data":"0e3b192c1d193f74865712c0ec501be652e2bb130c05cf2de9ec6f577e66384d"} Aug 04 20:11:27.321386 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:27.321313 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" podStartSLOduration=1.3212953 podStartE2EDuration="1.3212953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:11:27.319582751 +0000 UTC m=+3364.260379490" watchObservedRunningTime="2026-08-04 20:11:27.3212953 +0000 UTC m=+3364.262092046" Aug 04 20:11:36.910963 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:36.910917 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:36.910963 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:36.910976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:36.923516 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:36.923487 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:11:37.364602 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:11:37.364568 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:12:10.180708 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:10.180678 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw_e0d7260a-a0d4-431d-a994-78e876e6dc1c/payload-processing/0.log" Aug 04 20:12:15.173213 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.173173 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm"] Aug 04 20:12:15.176850 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.176831 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.179456 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.179436 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8f4e87\"" Aug 04 20:12:15.179529 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.179483 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8f4e87-dockercfg-mjt5c\"" Aug 04 20:12:15.186030 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.185997 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm"] Aug 04 20:12:15.216098 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.216054 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47"] Aug 04 20:12:15.219640 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.219617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.238610 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.238578 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47"] Aug 04 20:12:15.259280 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.259240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bb533c-c0bb-4852-80d8-71bfc7b10469-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.259480 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.259292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2fxf\" (UniqueName: \"kubernetes.io/projected/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-kube-api-access-s2fxf\") pod \"payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.259480 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.259450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.259576 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.259509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcsq\" (UniqueName: \"kubernetes.io/projected/05bb533c-c0bb-4852-80d8-71bfc7b10469-kube-api-access-zhcsq\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.360787 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.360694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcsq\" (UniqueName: \"kubernetes.io/projected/05bb533c-c0bb-4852-80d8-71bfc7b10469-kube-api-access-zhcsq\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.360787 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.360754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bb533c-c0bb-4852-80d8-71bfc7b10469-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.361041 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.360792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2fxf\" (UniqueName: \"kubernetes.io/projected/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-kube-api-access-s2fxf\") pod \"payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.361041 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.360876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.361573 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.361553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bb533c-c0bb-4852-80d8-71bfc7b10469-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.361623 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.361602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.369858 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.369833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2fxf\" (UniqueName: \"kubernetes.io/projected/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-kube-api-access-s2fxf\") pod \"payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.369858 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.369855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcsq\" (UniqueName: \"kubernetes.io/projected/05bb533c-c0bb-4852-80d8-71bfc7b10469-kube-api-access-zhcsq\") pod \"payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.490301 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.490207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:15.530730 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.530696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:15.653568 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.653534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm"] Aug 04 20:12:15.655172 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:12:15.655141 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bb533c_c0bb_4852_80d8_71bfc7b10469.slice/crio-a1f33106567070b1a1ab1238c2534755b37196db569186ce041d7d8de1777eeb WatchSource:0}: Error finding container a1f33106567070b1a1ab1238c2534755b37196db569186ce041d7d8de1777eeb: Status 404 returned error can't find the container with id a1f33106567070b1a1ab1238c2534755b37196db569186ce041d7d8de1777eeb Aug 04 20:12:15.699918 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:15.699888 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47"] Aug 04 20:12:15.712675 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:12:15.712640 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f7cc4_584b_4cdd_9fc3_eb3cd9ab4cf1.slice/crio-2fcc849ed809b7a1d5b0cd72c922419071719bfe2f503bce2cf18519df63801d WatchSource:0}: Error finding container 2fcc849ed809b7a1d5b0cd72c922419071719bfe2f503bce2cf18519df63801d: Status 404 returned error can't find the container with id 2fcc849ed809b7a1d5b0cd72c922419071719bfe2f503bce2cf18519df63801d Aug 04 20:12:16.556472 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.556429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" event={"ID":"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1","Type":"ContainerStarted","Data":"b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348"} Aug 04 20:12:16.556977 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.556503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" event={"ID":"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1","Type":"ContainerStarted","Data":"2fcc849ed809b7a1d5b0cd72c922419071719bfe2f503bce2cf18519df63801d"} Aug 04 20:12:16.556977 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.556556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:16.558258 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.558231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" event={"ID":"05bb533c-c0bb-4852-80d8-71bfc7b10469","Type":"ContainerStarted","Data":"54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43"} Aug 04 20:12:16.558402 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.558278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" event={"ID":"05bb533c-c0bb-4852-80d8-71bfc7b10469","Type":"ContainerStarted","Data":"a1f33106567070b1a1ab1238c2534755b37196db569186ce041d7d8de1777eeb"} Aug 04 20:12:16.558402 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.558309 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:16.572500 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.572450 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" podStartSLOduration=1.572433175 podStartE2EDuration="1.572433175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:12:16.570756407 +0000 UTC m=+3413.511553144" watchObservedRunningTime="2026-08-04 20:12:16.572433175 +0000 UTC m=+3413.513229997" Aug 04 20:12:16.586103 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:16.586053 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" podStartSLOduration=1.586035606 podStartE2EDuration="1.586035606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:12:16.585074574 +0000 UTC m=+3413.525871317" watchObservedRunningTime="2026-08-04 20:12:16.586035606 +0000 UTC m=+3413.526832384" Aug 04 20:12:27.566215 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:27.566172 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" podUID="05bb533c-c0bb-4852-80d8-71bfc7b10469" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.60:9004: connect: connection refused" Aug 04 20:12:27.566686 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:27.566497 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:37.564801 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:37.564764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:55.121635 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.121593 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47"] Aug 04 20:12:55.122122 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.121845 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" podUID="a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" containerName="payload-processing" containerID="cri-o://b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348" gracePeriod=30 Aug 04 20:12:55.135354 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.135290 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm"] Aug 04 20:12:55.135567 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.135526 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" podUID="05bb533c-c0bb-4852-80d8-71bfc7b10469" containerName="payload-pre-processing" containerID="cri-o://54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43" gracePeriod=30 Aug 04 20:12:55.405120 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.405094 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:55.445659 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.445628 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:55.462099 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.462009 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhcsq\" (UniqueName: \"kubernetes.io/projected/05bb533c-c0bb-4852-80d8-71bfc7b10469-kube-api-access-zhcsq\") pod \"05bb533c-c0bb-4852-80d8-71bfc7b10469\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " Aug 04 20:12:55.462099 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.462072 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bb533c-c0bb-4852-80d8-71bfc7b10469-plugins-config-volume\") pod \"05bb533c-c0bb-4852-80d8-71bfc7b10469\" (UID: \"05bb533c-c0bb-4852-80d8-71bfc7b10469\") " Aug 04 20:12:55.462368 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.462149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-plugins-config-volume\") pod \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " Aug 04 20:12:55.462575 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.462550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2fxf\" (UniqueName: \"kubernetes.io/projected/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-kube-api-access-s2fxf\") pod \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\" (UID: \"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1\") " Aug 04 20:12:55.463170 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.463108 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" (UID: "a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:12:55.464017 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.463977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bb533c-c0bb-4852-80d8-71bfc7b10469-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05bb533c-c0bb-4852-80d8-71bfc7b10469" (UID: "05bb533c-c0bb-4852-80d8-71bfc7b10469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:12:55.465128 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.465103 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bb533c-c0bb-4852-80d8-71bfc7b10469-kube-api-access-zhcsq" (OuterVolumeSpecName: "kube-api-access-zhcsq") pod "05bb533c-c0bb-4852-80d8-71bfc7b10469" (UID: "05bb533c-c0bb-4852-80d8-71bfc7b10469"). InnerVolumeSpecName "kube-api-access-zhcsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:12:55.465226 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.465193 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-kube-api-access-s2fxf" (OuterVolumeSpecName: "kube-api-access-s2fxf") pod "a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" (UID: "a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1"). InnerVolumeSpecName "kube-api-access-s2fxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:12:55.563543 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.563507 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcsq\" (UniqueName: \"kubernetes.io/projected/05bb533c-c0bb-4852-80d8-71bfc7b10469-kube-api-access-zhcsq\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.563543 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.563539 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05bb533c-c0bb-4852-80d8-71bfc7b10469-plugins-config-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.563543 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.563549 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-plugins-config-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.563796 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.563557 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2fxf\" (UniqueName: \"kubernetes.io/projected/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1-kube-api-access-s2fxf\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:12:55.752831 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.752730 2579 generic.go:356] "Generic (PLEG): container finished" podID="05bb533c-c0bb-4852-80d8-71bfc7b10469" containerID="54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43" exitCode=0 Aug 04 20:12:55.752831 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.752777 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" Aug 04 20:12:55.753062 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.752826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" event={"ID":"05bb533c-c0bb-4852-80d8-71bfc7b10469","Type":"ContainerDied","Data":"54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43"} Aug 04 20:12:55.753062 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.752877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm" event={"ID":"05bb533c-c0bb-4852-80d8-71bfc7b10469","Type":"ContainerDied","Data":"a1f33106567070b1a1ab1238c2534755b37196db569186ce041d7d8de1777eeb"} Aug 04 20:12:55.753062 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.752900 2579 scope.go:117] "RemoveContainer" containerID="54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43" Aug 04 20:12:55.754311 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.754281 2579 generic.go:356] "Generic (PLEG): container finished" podID="a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" containerID="b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348" exitCode=0 Aug 04 20:12:55.754450 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.754321 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" Aug 04 20:12:55.754450 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.754347 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" event={"ID":"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1","Type":"ContainerDied","Data":"b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348"} Aug 04 20:12:55.754450 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.754403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47" event={"ID":"a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1","Type":"ContainerDied","Data":"2fcc849ed809b7a1d5b0cd72c922419071719bfe2f503bce2cf18519df63801d"} Aug 04 20:12:55.762230 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.762211 2579 scope.go:117] "RemoveContainer" containerID="54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43" Aug 04 20:12:55.762572 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:12:55.762550 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43\": container with ID starting with 54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43 not found: ID does not exist" containerID="54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43" Aug 04 20:12:55.762679 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.762587 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43"} err="failed to get container status \"54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43\": rpc error: code = NotFound desc = could not find container \"54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43\": container with ID starting with 54405b276ce1303e6e9a5e1da7879c240a30a89dff296d38f593c5c0dc0e2f43 not found: ID does not exist" Aug 04 20:12:55.762679 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.762611 2579 scope.go:117] "RemoveContainer" containerID="b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348" Aug 04 20:12:55.771431 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.771409 2579 scope.go:117] "RemoveContainer" containerID="b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348" Aug 04 20:12:55.771771 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:12:55.771750 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348\": container with ID starting with b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348 not found: ID does not exist" containerID="b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348" Aug 04 20:12:55.771838 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.771780 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348"} err="failed to get container status \"b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348\": rpc error: code = NotFound desc = could not find container \"b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348\": container with ID starting with b6caf46348af112397c807b4a6ead2f541cc006e3c13d94fba10073a661e7348 not found: ID does not exist" Aug 04 20:12:55.773143 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.773124 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm"] Aug 04 20:12:55.776736 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.776713 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8f4e87-5c7b7b9648-hsmtm"] Aug 04 20:12:55.787460 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.787419 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47"] Aug 04 20:12:55.790872 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:55.790841 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8f4e87-c5ff95656-nrc47"] Aug 04 20:12:57.610396 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:57.610359 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05bb533c-c0bb-4852-80d8-71bfc7b10469" path="/var/lib/kubelet/pods/05bb533c-c0bb-4852-80d8-71bfc7b10469/volumes" Aug 04 20:12:57.610782 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:12:57.610752 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" path="/var/lib/kubelet/pods/a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1/volumes" Aug 04 20:13:05.246619 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.246575 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r"] Aug 04 20:13:05.247026 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.246991 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05bb533c-c0bb-4852-80d8-71bfc7b10469" containerName="payload-pre-processing" Aug 04 20:13:05.247026 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.247003 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05bb533c-c0bb-4852-80d8-71bfc7b10469" containerName="payload-pre-processing" Aug 04 20:13:05.247026 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.247015 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" containerName="payload-processing" Aug 04 20:13:05.247026 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.247021 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" containerName="payload-processing" Aug 04 20:13:05.247166 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.247079 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05bb533c-c0bb-4852-80d8-71bfc7b10469" containerName="payload-pre-processing" Aug 04 20:13:05.247166 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.247088 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39f7cc4-584b-4cdd-9fc3-eb3cd9ab4cf1" containerName="payload-processing" Aug 04 20:13:05.251455 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.251433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:05.254537 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.254510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 20:13:05.254688 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.254548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:13:05.261091 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.261063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r"] Aug 04 20:13:05.355446 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.355409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d193aba-6dcf-46e9-9659-30a16ef0ccee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r\" (UID: \"6d193aba-6dcf-46e9-9659-30a16ef0ccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:05.456935 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.456897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d193aba-6dcf-46e9-9659-30a16ef0ccee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r\" (UID: \"6d193aba-6dcf-46e9-9659-30a16ef0ccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:05.457633 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.457610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d193aba-6dcf-46e9-9659-30a16ef0ccee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r\" (UID: \"6d193aba-6dcf-46e9-9659-30a16ef0ccee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:05.563909 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.563866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:05.705516 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.705490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r"] Aug 04 20:13:05.708074 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:13:05.708049 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d193aba_6dcf_46e9_9659_30a16ef0ccee.slice/crio-c39e9955bef3d160306fe2858e7d5035a42807ccd880cd63e1f3e150014e974e WatchSource:0}: Error finding container c39e9955bef3d160306fe2858e7d5035a42807ccd880cd63e1f3e150014e974e: Status 404 returned error can't find the container with id c39e9955bef3d160306fe2858e7d5035a42807ccd880cd63e1f3e150014e974e Aug 04 20:13:05.810902 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:05.810864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" event={"ID":"6d193aba-6dcf-46e9-9659-30a16ef0ccee","Type":"ContainerStarted","Data":"c39e9955bef3d160306fe2858e7d5035a42807ccd880cd63e1f3e150014e974e"} Aug 04 20:13:08.828742 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:08.828708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" event={"ID":"6d193aba-6dcf-46e9-9659-30a16ef0ccee","Type":"ContainerStarted","Data":"ac3fd12a058fe6f2ee2f14cc749aa5ce8f7fe9ea6910e1965b2eeb6387035cb9"} Aug 04 20:13:08.844518 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:08.844464 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" podStartSLOduration=1.436734612 podStartE2EDuration="3.844441274s" totalImagesPullingTime="2.407706662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 20:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:13:08.842897818 +0000 UTC m=+3465.783694565" watchObservedRunningTime="2026-08-04 20:13:08.844441274 +0000 UTC m=+3465.785238021" Aug 04 20:13:09.835233 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:09.835199 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d193aba-6dcf-46e9-9659-30a16ef0ccee" containerID="ac3fd12a058fe6f2ee2f14cc749aa5ce8f7fe9ea6910e1965b2eeb6387035cb9" exitCode=0 Aug 04 20:13:09.835646 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:09.835281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" event={"ID":"6d193aba-6dcf-46e9-9659-30a16ef0ccee","Type":"ContainerDied","Data":"ac3fd12a058fe6f2ee2f14cc749aa5ce8f7fe9ea6910e1965b2eeb6387035cb9"} Aug 04 20:13:10.980071 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:10.980044 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:11.111397 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:11.111293 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d193aba-6dcf-46e9-9659-30a16ef0ccee-maas-api-service-ca\") pod \"6d193aba-6dcf-46e9-9659-30a16ef0ccee\" (UID: \"6d193aba-6dcf-46e9-9659-30a16ef0ccee\") " Aug 04 20:13:11.111635 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:11.111610 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d193aba-6dcf-46e9-9659-30a16ef0ccee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d193aba-6dcf-46e9-9659-30a16ef0ccee" (UID: "6d193aba-6dcf-46e9-9659-30a16ef0ccee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:13:11.212501 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:11.212457 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d193aba-6dcf-46e9-9659-30a16ef0ccee-maas-api-service-ca\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:11.846713 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:11.846683 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" Aug 04 20:13:11.846894 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:11.846686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r" event={"ID":"6d193aba-6dcf-46e9-9659-30a16ef0ccee","Type":"ContainerDied","Data":"c39e9955bef3d160306fe2858e7d5035a42807ccd880cd63e1f3e150014e974e"} Aug 04 20:13:11.846894 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:11.846810 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39e9955bef3d160306fe2858e7d5035a42807ccd880cd63e1f3e150014e974e" Aug 04 20:13:20.945525 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:20.945454 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r"] Aug 04 20:13:20.948063 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:20.948036 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-41ab60-rhm6r"] Aug 04 20:13:21.610594 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:21.610550 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d193aba-6dcf-46e9-9659-30a16ef0ccee" path="/var/lib/kubelet/pods/6d193aba-6dcf-46e9-9659-30a16ef0ccee/volumes" Aug 04 20:13:26.302155 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:26.302108 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb"] Aug 04 20:13:41.322114 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.322067 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" podUID="34d86966-aef0-4b84-b131-c1613a54a2ba" containerName="main" containerID="cri-o://3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249" gracePeriod=45 Aug 04 20:13:41.581840 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.581765 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:13:41.604897 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.604558 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-tmp-dir\") pod \"34d86966-aef0-4b84-b131-c1613a54a2ba\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " Aug 04 20:13:41.604897 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.604851 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34d86966-aef0-4b84-b131-c1613a54a2ba" (UID: "34d86966-aef0-4b84-b131-c1613a54a2ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:41.605773 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.605414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksvj5\" (UniqueName: \"kubernetes.io/projected/34d86966-aef0-4b84-b131-c1613a54a2ba-kube-api-access-ksvj5\") pod \"34d86966-aef0-4b84-b131-c1613a54a2ba\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " Aug 04 20:13:41.605773 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.605461 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-dshm\") pod \"34d86966-aef0-4b84-b131-c1613a54a2ba\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " Aug 04 20:13:41.605773 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.605537 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-home\") pod \"34d86966-aef0-4b84-b131-c1613a54a2ba\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " Aug 04 20:13:41.605773 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.605581 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d86966-aef0-4b84-b131-c1613a54a2ba-tls-certs\") pod \"34d86966-aef0-4b84-b131-c1613a54a2ba\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " Aug 04 20:13:41.605773 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.605658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-model-cache\") pod \"34d86966-aef0-4b84-b131-c1613a54a2ba\" (UID: \"34d86966-aef0-4b84-b131-c1613a54a2ba\") " Aug 04 20:13:41.606191 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.606036 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-tmp-dir\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:41.606407 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.606284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-home" (OuterVolumeSpecName: "home") pod "34d86966-aef0-4b84-b131-c1613a54a2ba" (UID: "34d86966-aef0-4b84-b131-c1613a54a2ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:41.606945 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.606239 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "34d86966-aef0-4b84-b131-c1613a54a2ba" (UID: "34d86966-aef0-4b84-b131-c1613a54a2ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:41.609700 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.609621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d86966-aef0-4b84-b131-c1613a54a2ba-kube-api-access-ksvj5" (OuterVolumeSpecName: "kube-api-access-ksvj5") pod "34d86966-aef0-4b84-b131-c1613a54a2ba" (UID: "34d86966-aef0-4b84-b131-c1613a54a2ba"). InnerVolumeSpecName "kube-api-access-ksvj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:13:41.609700 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.609641 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d86966-aef0-4b84-b131-c1613a54a2ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34d86966-aef0-4b84-b131-c1613a54a2ba" (UID: "34d86966-aef0-4b84-b131-c1613a54a2ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 20:13:41.609700 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.609665 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-dshm" (OuterVolumeSpecName: "dshm") pod "34d86966-aef0-4b84-b131-c1613a54a2ba" (UID: "34d86966-aef0-4b84-b131-c1613a54a2ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:13:41.707532 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.707498 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-model-cache\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:41.707725 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.707563 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvj5\" (UniqueName: \"kubernetes.io/projected/34d86966-aef0-4b84-b131-c1613a54a2ba-kube-api-access-ksvj5\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:41.707725 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.707581 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-dshm\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:41.707725 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.707602 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d86966-aef0-4b84-b131-c1613a54a2ba-home\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:41.707725 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:41.707613 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d86966-aef0-4b84-b131-c1613a54a2ba-tls-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:13:42.001884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.001791 2579 generic.go:356] "Generic (PLEG): container finished" podID="34d86966-aef0-4b84-b131-c1613a54a2ba" containerID="3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249" exitCode=0 Aug 04 20:13:42.001884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.001833 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" Aug 04 20:13:42.002082 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.001879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" event={"ID":"34d86966-aef0-4b84-b131-c1613a54a2ba","Type":"ContainerDied","Data":"3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249"} Aug 04 20:13:42.002082 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.001921 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb" event={"ID":"34d86966-aef0-4b84-b131-c1613a54a2ba","Type":"ContainerDied","Data":"0e3b192c1d193f74865712c0ec501be652e2bb130c05cf2de9ec6f577e66384d"} Aug 04 20:13:42.002082 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.001939 2579 scope.go:117] "RemoveContainer" containerID="3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249" Aug 04 20:13:42.010851 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.010825 2579 scope.go:117] "RemoveContainer" containerID="3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249" Aug 04 20:13:42.011134 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:13:42.011112 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249\": container with ID starting with 3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249 not found: ID does not exist" containerID="3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249" Aug 04 20:13:42.011217 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.011146 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249"} err="failed to get container status \"3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249\": rpc error: code = NotFound desc = could not find container \"3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249\": container with ID starting with 3326b00f7617ac87d7cada13af9351fa83c680093270adf1f21ea5681a555249 not found: ID does not exist" Aug 04 20:13:42.017131 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.017099 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb"] Aug 04 20:13:42.023291 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:42.023261 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-41ab60/ipp-route-41ab60-kserve-d5b5c8cff-zcwwb"] Aug 04 20:13:43.611047 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:13:43.611009 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d86966-aef0-4b84-b131-c1613a54a2ba" path="/var/lib/kubelet/pods/34d86966-aef0-4b84-b131-c1613a54a2ba/volumes" Aug 04 20:14:09.559839 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.559805 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz"] Aug 04 20:14:09.560341 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.560190 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d193aba-6dcf-46e9-9659-30a16ef0ccee" containerName="revoke-keys" Aug 04 20:14:09.560341 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.560201 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d193aba-6dcf-46e9-9659-30a16ef0ccee" containerName="revoke-keys" Aug 04 20:14:09.560341 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.560216 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d86966-aef0-4b84-b131-c1613a54a2ba" containerName="main" Aug 04 20:14:09.560341 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.560223 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d86966-aef0-4b84-b131-c1613a54a2ba" containerName="main" Aug 04 20:14:09.560341 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.560290 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d86966-aef0-4b84-b131-c1613a54a2ba" containerName="main" Aug 04 20:14:09.560341 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.560301 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d193aba-6dcf-46e9-9659-30a16ef0ccee" containerName="revoke-keys" Aug 04 20:14:09.563415 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.563396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:09.565920 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.565900 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:14:09.566212 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.566194 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 20:14:09.570870 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.570842 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz"] Aug 04 20:14:09.679539 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.679497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/687c9504-ab22-4a59-8ac3-af255abcdaad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz\" (UID: \"687c9504-ab22-4a59-8ac3-af255abcdaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:09.780960 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.780924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/687c9504-ab22-4a59-8ac3-af255abcdaad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz\" (UID: \"687c9504-ab22-4a59-8ac3-af255abcdaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:09.781635 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.781616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/687c9504-ab22-4a59-8ac3-af255abcdaad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz\" (UID: \"687c9504-ab22-4a59-8ac3-af255abcdaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:09.876808 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:09.876712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:10.020261 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:10.020233 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz"] Aug 04 20:14:10.022190 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:14:10.022157 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687c9504_ab22_4a59_8ac3_af255abcdaad.slice/crio-3bcdf960343d23288f4cfd6b0c7c0d8aae4a33167179cc5076ada4d445023c9f WatchSource:0}: Error finding container 3bcdf960343d23288f4cfd6b0c7c0d8aae4a33167179cc5076ada4d445023c9f: Status 404 returned error can't find the container with id 3bcdf960343d23288f4cfd6b0c7c0d8aae4a33167179cc5076ada4d445023c9f Aug 04 20:14:10.143391 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:10.143265 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" event={"ID":"687c9504-ab22-4a59-8ac3-af255abcdaad","Type":"ContainerStarted","Data":"3bcdf960343d23288f4cfd6b0c7c0d8aae4a33167179cc5076ada4d445023c9f"} Aug 04 20:14:11.149922 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:11.149828 2579 generic.go:356] "Generic (PLEG): container finished" podID="687c9504-ab22-4a59-8ac3-af255abcdaad" containerID="6d1673ca54cd55df31ec063a046116b06e0d95cf974a88273f9266bd6e3f5fec" exitCode=0 Aug 04 20:14:11.149922 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:11.149904 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" event={"ID":"687c9504-ab22-4a59-8ac3-af255abcdaad","Type":"ContainerDied","Data":"6d1673ca54cd55df31ec063a046116b06e0d95cf974a88273f9266bd6e3f5fec"} Aug 04 20:14:12.299637 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:12.299612 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:12.303535 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:12.303509 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/687c9504-ab22-4a59-8ac3-af255abcdaad-maas-api-service-ca\") pod \"687c9504-ab22-4a59-8ac3-af255abcdaad\" (UID: \"687c9504-ab22-4a59-8ac3-af255abcdaad\") " Aug 04 20:14:12.303842 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:12.303822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687c9504-ab22-4a59-8ac3-af255abcdaad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "687c9504-ab22-4a59-8ac3-af255abcdaad" (UID: "687c9504-ab22-4a59-8ac3-af255abcdaad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:12.404962 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:12.404924 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/687c9504-ab22-4a59-8ac3-af255abcdaad-maas-api-service-ca\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:13.162976 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:13.162942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" event={"ID":"687c9504-ab22-4a59-8ac3-af255abcdaad","Type":"ContainerDied","Data":"3bcdf960343d23288f4cfd6b0c7c0d8aae4a33167179cc5076ada4d445023c9f"} Aug 04 20:14:13.163151 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:13.162999 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bcdf960343d23288f4cfd6b0c7c0d8aae4a33167179cc5076ada4d445023c9f" Aug 04 20:14:13.163151 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:13.162956 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz" Aug 04 20:14:20.708197 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:20.708160 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw"] Aug 04 20:14:20.708639 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:20.708481 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" podUID="e0d7260a-a0d4-431d-a994-78e876e6dc1c" containerName="payload-processing" containerID="cri-o://32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b" gracePeriod=30 Aug 04 20:14:20.980463 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:20.980436 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:14:21.012749 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.012697 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz"] Aug 04 20:14:21.014942 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.014910 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9cd851-sfdrz"] Aug 04 20:14:21.081547 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.081507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7260a-a0d4-431d-a994-78e876e6dc1c-plugins-config-volume\") pod \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " Aug 04 20:14:21.081758 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.081562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqjft\" (UniqueName: \"kubernetes.io/projected/e0d7260a-a0d4-431d-a994-78e876e6dc1c-kube-api-access-rqjft\") pod \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\" (UID: \"e0d7260a-a0d4-431d-a994-78e876e6dc1c\") " Aug 04 20:14:21.081938 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.081910 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d7260a-a0d4-431d-a994-78e876e6dc1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0d7260a-a0d4-431d-a994-78e876e6dc1c" (UID: "e0d7260a-a0d4-431d-a994-78e876e6dc1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:21.083844 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.083814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d7260a-a0d4-431d-a994-78e876e6dc1c-kube-api-access-rqjft" (OuterVolumeSpecName: "kube-api-access-rqjft") pod "e0d7260a-a0d4-431d-a994-78e876e6dc1c" (UID: "e0d7260a-a0d4-431d-a994-78e876e6dc1c"). InnerVolumeSpecName "kube-api-access-rqjft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:21.182520 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.182469 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7260a-a0d4-431d-a994-78e876e6dc1c-plugins-config-volume\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.182520 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.182512 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqjft\" (UniqueName: \"kubernetes.io/projected/e0d7260a-a0d4-431d-a994-78e876e6dc1c-kube-api-access-rqjft\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:21.206269 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.206239 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0d7260a-a0d4-431d-a994-78e876e6dc1c" containerID="32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b" exitCode=0 Aug 04 20:14:21.206456 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.206284 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" Aug 04 20:14:21.206456 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.206339 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" event={"ID":"e0d7260a-a0d4-431d-a994-78e876e6dc1c","Type":"ContainerDied","Data":"32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b"} Aug 04 20:14:21.206456 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.206395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw" event={"ID":"e0d7260a-a0d4-431d-a994-78e876e6dc1c","Type":"ContainerDied","Data":"103411358eb1d9cd0af13b53eed43dcbb8283578340a124c648a487237134838"} Aug 04 20:14:21.206456 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.206418 2579 scope.go:117] "RemoveContainer" containerID="32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b" Aug 04 20:14:21.215700 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.215668 2579 scope.go:117] "RemoveContainer" containerID="32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b" Aug 04 20:14:21.216002 ip-10-0-143-62 kubenswrapper[2579]: E0804 20:14:21.215982 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b\": container with ID starting with 32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b not found: ID does not exist" containerID="32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b" Aug 04 20:14:21.216073 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.216016 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b"} err="failed to get container status \"32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b\": rpc error: code = NotFound desc = could not find container \"32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b\": container with ID starting with 32495e60594048b9c746533ae13dfc5f6e681de73dd17ef0c519e43118e8040b not found: ID does not exist" Aug 04 20:14:21.230890 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.230857 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw"] Aug 04 20:14:21.237522 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.237489 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9cd851-5d8c64c868-qm5fw"] Aug 04 20:14:21.611198 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.611159 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687c9504-ab22-4a59-8ac3-af255abcdaad" path="/var/lib/kubelet/pods/687c9504-ab22-4a59-8ac3-af255abcdaad/volumes" Aug 04 20:14:21.611591 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:21.611577 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d7260a-a0d4-431d-a994-78e876e6dc1c" path="/var/lib/kubelet/pods/e0d7260a-a0d4-431d-a994-78e876e6dc1c/volumes" Aug 04 20:14:27.920587 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.920551 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz"] Aug 04 20:14:27.920993 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.920912 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d7260a-a0d4-431d-a994-78e876e6dc1c" containerName="payload-processing" Aug 04 20:14:27.920993 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.920924 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d7260a-a0d4-431d-a994-78e876e6dc1c" containerName="payload-processing" Aug 04 20:14:27.920993 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.920942 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687c9504-ab22-4a59-8ac3-af255abcdaad" containerName="revoke-keys" Aug 04 20:14:27.920993 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.920948 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="687c9504-ab22-4a59-8ac3-af255abcdaad" containerName="revoke-keys" Aug 04 20:14:27.921126 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.921017 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d7260a-a0d4-431d-a994-78e876e6dc1c" containerName="payload-processing" Aug 04 20:14:27.921126 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.921028 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="687c9504-ab22-4a59-8ac3-af255abcdaad" containerName="revoke-keys" Aug 04 20:14:27.925481 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.925458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:27.928405 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.928376 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 20:14:27.928652 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.928386 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bp7tl\"" Aug 04 20:14:27.931856 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:27.931824 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz"] Aug 04 20:14:28.049767 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:28.049725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3476c40-b72d-402b-8834-4f92069e0220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz\" (UID: \"f3476c40-b72d-402b-8834-4f92069e0220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:28.151019 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:28.150976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3476c40-b72d-402b-8834-4f92069e0220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz\" (UID: \"f3476c40-b72d-402b-8834-4f92069e0220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:28.151678 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:28.151656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3476c40-b72d-402b-8834-4f92069e0220-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz\" (UID: \"f3476c40-b72d-402b-8834-4f92069e0220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:28.238015 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:28.237919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:28.377616 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:28.377590 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz"] Aug 04 20:14:28.380349 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:14:28.380297 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3476c40_b72d_402b_8834_4f92069e0220.slice/crio-2fd7036c58b09175c3c42790069a9119b1cfd604c6d8d0c47b11b96792aacb33 WatchSource:0}: Error finding container 2fd7036c58b09175c3c42790069a9119b1cfd604c6d8d0c47b11b96792aacb33: Status 404 returned error can't find the container with id 2fd7036c58b09175c3c42790069a9119b1cfd604c6d8d0c47b11b96792aacb33 Aug 04 20:14:29.248345 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:29.248293 2579 generic.go:356] "Generic (PLEG): container finished" podID="f3476c40-b72d-402b-8834-4f92069e0220" containerID="3a900c7636d516bd8014616ceffb245539c4dd188238aca510724f5cc9240158" exitCode=0 Aug 04 20:14:29.248734 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:29.248395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" event={"ID":"f3476c40-b72d-402b-8834-4f92069e0220","Type":"ContainerDied","Data":"3a900c7636d516bd8014616ceffb245539c4dd188238aca510724f5cc9240158"} Aug 04 20:14:29.248734 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:29.248443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" event={"ID":"f3476c40-b72d-402b-8834-4f92069e0220","Type":"ContainerStarted","Data":"2fd7036c58b09175c3c42790069a9119b1cfd604c6d8d0c47b11b96792aacb33"} Aug 04 20:14:30.397940 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:30.397913 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:30.475155 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:30.475119 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3476c40-b72d-402b-8834-4f92069e0220-maas-api-service-ca\") pod \"f3476c40-b72d-402b-8834-4f92069e0220\" (UID: \"f3476c40-b72d-402b-8834-4f92069e0220\") " Aug 04 20:14:30.475544 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:30.475515 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3476c40-b72d-402b-8834-4f92069e0220-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3476c40-b72d-402b-8834-4f92069e0220" (UID: "f3476c40-b72d-402b-8834-4f92069e0220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:30.576499 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:30.576460 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3476c40-b72d-402b-8834-4f92069e0220-maas-api-service-ca\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:31.259778 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:31.259743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" event={"ID":"f3476c40-b72d-402b-8834-4f92069e0220","Type":"ContainerDied","Data":"2fd7036c58b09175c3c42790069a9119b1cfd604c6d8d0c47b11b96792aacb33"} Aug 04 20:14:31.259778 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:31.259778 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz" Aug 04 20:14:31.259992 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:31.259800 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fd7036c58b09175c3c42790069a9119b1cfd604c6d8d0c47b11b96792aacb33" Aug 04 20:14:38.644553 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:38.644513 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz"] Aug 04 20:14:38.647799 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:38.647764 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-03ae76-h9xmz"] Aug 04 20:14:39.610777 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:39.610746 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3476c40-b72d-402b-8834-4f92069e0220" path="/var/lib/kubelet/pods/f3476c40-b72d-402b-8834-4f92069e0220/volumes" Aug 04 20:14:45.157743 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:45.157702 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5"] Aug 04 20:14:45.158403 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:45.157933 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" containerID="cri-o://49038576108ccd8afa63ed38e75d7154702cb9493562ae3c87ee97e429363294" gracePeriod=30 Aug 04 20:14:49.538344 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:49.538298 2579 patch_prober.go:28] interesting pod/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 20:14:49.538727 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:49.538384 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 20:14:50.357624 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.357587 2579 generic.go:356] "Generic (PLEG): container finished" podID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerID="49038576108ccd8afa63ed38e75d7154702cb9493562ae3c87ee97e429363294" exitCode=0 Aug 04 20:14:50.357795 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.357656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" event={"ID":"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d","Type":"ContainerDied","Data":"49038576108ccd8afa63ed38e75d7154702cb9493562ae3c87ee97e429363294"} Aug 04 20:14:50.357795 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.357719 2579 scope.go:117] "RemoveContainer" containerID="779f2a21c687de3c9935628d074866d00b2478f96c59a7b22fdf22eccdcf4f50" Aug 04 20:14:50.419861 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.419834 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 20:14:50.570039 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.569999 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istiod-ca-cert\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570039 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570047 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-credential-socket\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570076 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-podinfo\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570124 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-token\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570151 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-data\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-socket\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570237 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-envoy\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtqgq\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-kube-api-access-gtqgq\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-certs\") pod \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\" (UID: \"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d\") " Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570413 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570494 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:14:50.570611 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570603 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:50.571153 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570754 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:50.571153 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.570908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:50.571274 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.571190 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.571274 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.571214 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.571274 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.571230 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.571274 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.571244 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.571274 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.571258 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.572630 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.572600 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:50.572786 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.572761 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 20:14:50.572851 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.572767 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-kube-api-access-gtqgq" (OuterVolumeSpecName: "kube-api-access-gtqgq") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "kube-api-access-gtqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:14:50.572851 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.572790 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" (UID: "8b9e0f86-4b81-49a3-9e58-dae1b91eb65d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:14:50.672002 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.671958 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.672002 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.671996 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.672002 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.672005 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:50.672255 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:50.672014 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtqgq\" (UniqueName: \"kubernetes.io/projected/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d-kube-api-access-gtqgq\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:14:51.364286 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:51.364249 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" Aug 04 20:14:51.364464 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:51.364244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5" event={"ID":"8b9e0f86-4b81-49a3-9e58-dae1b91eb65d","Type":"ContainerDied","Data":"b11cd45ec002f929f93692369a2746cb583a9c3e2ac3327767b6dac54974f9c3"} Aug 04 20:14:51.364464 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:51.364395 2579 scope.go:117] "RemoveContainer" containerID="49038576108ccd8afa63ed38e75d7154702cb9493562ae3c87ee97e429363294" Aug 04 20:14:51.390149 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:51.390105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5"] Aug 04 20:14:51.394063 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:51.394033 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-03ae76-openshift-default-85f656f8d8-pgnk5"] Aug 04 20:14:51.610547 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:51.610509 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" path="/var/lib/kubelet/pods/8b9e0f86-4b81-49a3-9e58-dae1b91eb65d/volumes" Aug 04 20:14:56.157056 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:56.157018 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st"] Aug 04 20:14:56.157599 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:14:56.157291 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" containerID="cri-o://b8bab173df208a6c26d64ea73865df51194bacf3c50b8b5ad9302c2cad550063" gracePeriod=30 Aug 04 20:15:01.417108 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.417077 2579 generic.go:356] "Generic (PLEG): container finished" podID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerID="b8bab173df208a6c26d64ea73865df51194bacf3c50b8b5ad9302c2cad550063" exitCode=0 Aug 04 20:15:01.417490 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.417144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" event={"ID":"468d6e7b-a855-4c97-9f46-ec8b36d237b8","Type":"ContainerDied","Data":"b8bab173df208a6c26d64ea73865df51194bacf3c50b8b5ad9302c2cad550063"} Aug 04 20:15:01.417490 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.417207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" event={"ID":"468d6e7b-a855-4c97-9f46-ec8b36d237b8","Type":"ContainerDied","Data":"2fad920d0323901417a85d87cb2ca801fdad88dafc6886b1f17d54bca64b9b3d"} Aug 04 20:15:01.417490 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.417225 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fad920d0323901417a85d87cb2ca801fdad88dafc6886b1f17d54bca64b9b3d" Aug 04 20:15:01.417490 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.417238 2579 scope.go:117] "RemoveContainer" containerID="b0ee25bf8d662cd72d52a92cf2bd3b4fa8257f1e51e37769f77c459398bb1d4c" Aug 04 20:15:01.421955 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.421880 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:15:01.465814 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.465779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-envoy\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.465972 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.465840 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istiod-ca-cert\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.465972 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.465875 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-data\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.465972 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.465927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m769x\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-kube-api-access-m769x\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.466157 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.465973 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-credential-socket\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.466157 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466018 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-socket\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.466262 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466239 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:15:01.466402 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:15:01.466402 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466309 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:15:01.466402 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466383 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 20:15:01.466402 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466393 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-token\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.466586 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466431 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-podinfo\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.466586 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-certs\") pod \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\" (UID: \"468d6e7b-a855-4c97-9f46-ec8b36d237b8\") " Aug 04 20:15:01.466770 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466744 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:15:01.466895 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466879 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.466953 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466901 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-workload-certs\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.466953 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466915 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istiod-ca-cert\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.466953 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466927 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-data\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.466953 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.466941 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-credential-socket\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.468465 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.468433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 20:15:01.468685 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.468667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 20:15:01.468753 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.468729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:15:01.468815 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.468729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-kube-api-access-m769x" (OuterVolumeSpecName: "kube-api-access-m769x") pod "468d6e7b-a855-4c97-9f46-ec8b36d237b8" (UID: "468d6e7b-a855-4c97-9f46-ec8b36d237b8"). InnerVolumeSpecName "kube-api-access-m769x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 20:15:01.567659 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.567615 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-token\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.567659 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.567650 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-podinfo\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.567659 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.567663 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468d6e7b-a855-4c97-9f46-ec8b36d237b8-istio-envoy\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:01.567906 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:01.567674 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m769x\" (UniqueName: \"kubernetes.io/projected/468d6e7b-a855-4c97-9f46-ec8b36d237b8-kube-api-access-m769x\") on node \"ip-10-0-143-62.ec2.internal\" DevicePath \"\"" Aug 04 20:15:02.423457 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:02.423429 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st" Aug 04 20:15:02.457555 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:02.457510 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st"] Aug 04 20:15:02.461212 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:02.461179 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-15a874-openshift-default-85ddb9f868-f65st"] Aug 04 20:15:02.770158 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:02.770076 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-ljzlw_1de709ab-960b-4bcc-a7b9-58a687572868/copy-manifests/0.log" Aug 04 20:15:02.789031 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:02.788994 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-ljzlw_1de709ab-960b-4bcc-a7b9-58a687572868/manager/0.log" Aug 04 20:15:03.610622 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:03.610587 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" path="/var/lib/kubelet/pods/468d6e7b-a855-4c97-9f46-ec8b36d237b8/volumes" Aug 04 20:15:04.643739 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:04.643703 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-68kqs_da8d7e93-d70b-401f-a5ba-d302696cd9ef/manager/0.log" Aug 04 20:15:04.870852 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:04.870816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-gfxkx_71660ffb-f55f-457d-8fad-fc37bb9fd937/manager/0.log" Aug 04 20:15:05.440505 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:05.440471 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4_0f0bff41-45c7-4708-a703-b88f946d6da3/istio-proxy/0.log" Aug 04 20:15:05.891466 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:05.891430 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/1.log" Aug 04 20:15:06.230945 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:06.230863 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd5c49894-ktdqh_c9efa644-f790-4e4b-8ddf-a9a75d1c9719/router/0.log" Aug 04 20:15:06.673666 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:06.673632 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2n25n_9868c8f4-af52-4b34-8c75-97096e41a217/main/0.log" Aug 04 20:15:06.894351 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:06.894295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68flkw6n_e0b10b9f-6c58-409b-92bd-5272f14d5e57/main/0.log" Aug 04 20:15:07.011209 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:07.011122 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-czsqm_3c16efa8-01a3-4722-9703-fc93b9de8cb6/main/0.log" Aug 04 20:15:15.776165 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:15.776080 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-glb96_4c10f094-a78b-4733-b541-57206eefdc6f/global-pull-secret-syncer/0.log" Aug 04 20:15:15.962598 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:15.962564 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9f99v_a6a265ca-059b-46f3-b9d9-a0589199d67c/konnectivity-agent/0.log" Aug 04 20:15:16.033296 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:16.033263 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-62.ec2.internal_76dbb7836f721215f5525cdef4bf64f8/haproxy/0.log" Aug 04 20:15:20.999984 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:20.999928 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-68kqs_da8d7e93-d70b-401f-a5ba-d302696cd9ef/manager/0.log" Aug 04 20:15:21.191303 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:21.191258 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-gfxkx_71660ffb-f55f-457d-8fad-fc37bb9fd937/manager/0.log" Aug 04 20:15:22.891056 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:22.891013 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-ngdn9_6f6e778e-c311-4bab-b5f4-b4614e3b7795/cluster-monitoring-operator/0.log" Aug 04 20:15:23.402698 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:23.402660 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9fjf_c3258235-b5d4-413b-b31f-a589c2b45a25/node-exporter/0.log" Aug 04 20:15:23.428174 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:23.428145 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9fjf_c3258235-b5d4-413b-b31f-a589c2b45a25/kube-rbac-proxy/0.log" Aug 04 20:15:23.450603 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:23.450572 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9fjf_c3258235-b5d4-413b-b31f-a589c2b45a25/init-textfile/0.log" Aug 04 20:15:23.863275 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:23.863154 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:15:23.871311 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:23.868699 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:15:24.077323 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:24.077292 2579 scope.go:117] "RemoveContainer" containerID="b8bab173df208a6c26d64ea73865df51194bacf3c50b8b5ad9302c2cad550063" Aug 04 20:15:25.155781 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.155751 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-x7ktf_2709f187-9682-4546-ba3f-b8671e3f2ab7/networking-console-plugin/0.log" Aug 04 20:15:25.620574 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620541 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc"] Aug 04 20:15:25.620897 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620882 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3476c40-b72d-402b-8834-4f92069e0220" containerName="revoke-keys" Aug 04 20:15:25.620943 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620899 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3476c40-b72d-402b-8834-4f92069e0220" containerName="revoke-keys" Aug 04 20:15:25.620943 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620914 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" Aug 04 20:15:25.620943 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620919 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" Aug 04 20:15:25.620943 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620931 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" Aug 04 20:15:25.620943 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" Aug 04 20:15:25.620943 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620945 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620950 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620959 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.620964 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.621023 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3476c40-b72d-402b-8834-4f92069e0220" containerName="revoke-keys" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.621031 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9e0f86-4b81-49a3-9e58-dae1b91eb65d" containerName="istio-proxy" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.621037 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" Aug 04 20:15:25.621135 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.621044 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="468d6e7b-a855-4c97-9f46-ec8b36d237b8" containerName="istio-proxy" Aug 04 20:15:25.625240 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.625220 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.627512 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.627485 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bk9n8\"/\"default-dockercfg-qgnpn\"" Aug 04 20:15:25.627664 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.627493 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bk9n8\"/\"kube-root-ca.crt\"" Aug 04 20:15:25.628319 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.628297 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bk9n8\"/\"openshift-service-ca.crt\"" Aug 04 20:15:25.637237 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.637211 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc"] Aug 04 20:15:25.691016 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.690962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-proc\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.691229 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.691049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-sys\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.691229 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.691070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5mg\" (UniqueName: \"kubernetes.io/projected/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-kube-api-access-7p5mg\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.691229 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.691112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-podres\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.691229 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.691175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-lib-modules\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.770109 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.770078 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/1.log" Aug 04 20:15:25.778843 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.778816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-cgnz7_47ff5264-2f50-43ed-ad20-820410bb85c7/console-operator/2.log" Aug 04 20:15:25.792496 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-proc\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792618 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-sys\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792618 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5mg\" (UniqueName: \"kubernetes.io/projected/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-kube-api-access-7p5mg\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792618 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-podres\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792618 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-lib-modules\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792753 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-sys\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792753 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-proc\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792753 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-podres\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.792753 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.792732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-lib-modules\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.800620 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.800567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5mg\" (UniqueName: \"kubernetes.io/projected/755c51e8-406a-46d1-a9dd-69f1aeddaa1a-kube-api-access-7p5mg\") pod \"perf-node-gather-daemonset-6csrc\" (UID: \"755c51e8-406a-46d1-a9dd-69f1aeddaa1a\") " pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:25.936048 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:25.935946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:26.072667 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:26.072615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc"] Aug 04 20:15:26.075035 ip-10-0-143-62 kubenswrapper[2579]: W0804 20:15:26.075001 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod755c51e8_406a_46d1_a9dd_69f1aeddaa1a.slice/crio-8cd0055034f97d40107f05f961bdd8e99f088bec7e93dc98a906c47f3e5ca8e7 WatchSource:0}: Error finding container 8cd0055034f97d40107f05f961bdd8e99f088bec7e93dc98a906c47f3e5ca8e7: Status 404 returned error can't find the container with id 8cd0055034f97d40107f05f961bdd8e99f088bec7e93dc98a906c47f3e5ca8e7 Aug 04 20:15:26.367451 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:26.367415 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-vtn4d_2451817b-9161-4742-bcbf-1cc4ebb5070e/download-server/0.log" Aug 04 20:15:26.548575 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:26.548539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" event={"ID":"755c51e8-406a-46d1-a9dd-69f1aeddaa1a","Type":"ContainerStarted","Data":"5077d161fda1c66f513b2334ce04c938ebb97c4c0829c7b6d0c1dc07a730ff5b"} Aug 04 20:15:26.548777 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:26.548597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" event={"ID":"755c51e8-406a-46d1-a9dd-69f1aeddaa1a","Type":"ContainerStarted","Data":"8cd0055034f97d40107f05f961bdd8e99f088bec7e93dc98a906c47f3e5ca8e7"} Aug 04 20:15:26.548777 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:26.548630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:26.564703 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:26.564650 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" podStartSLOduration=1.564633771 podStartE2EDuration="1.564633771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 20:15:26.562642791 +0000 UTC m=+3603.503439537" watchObservedRunningTime="2026-08-04 20:15:26.564633771 +0000 UTC m=+3603.505430517" Aug 04 20:15:27.039278 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:27.039242 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-cpb76_5d58d284-9cd3-40f7-bc60-3116bc0cc32f/volume-data-source-validator/0.log" Aug 04 20:15:28.052968 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:28.052939 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p589j_6d50fb28-2b17-4db2-8465-dc86997e17ec/dns/0.log" Aug 04 20:15:28.072701 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:28.072668 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p589j_6d50fb28-2b17-4db2-8465-dc86997e17ec/kube-rbac-proxy/0.log" Aug 04 20:15:28.147319 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:28.147288 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b6mw2_258893ce-f863-4989-bfb6-2076f423bad4/dns-node-resolver/0.log" Aug 04 20:15:28.730439 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:28.730406 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4sdzl_d27cb5fd-9a5d-4afa-b3c1-177a83080e08/node-ca/0.log" Aug 04 20:15:29.672373 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:29.672317 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4_0f0bff41-45c7-4708-a703-b88f946d6da3/istio-proxy/0.log" Aug 04 20:15:30.096879 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:30.096849 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/0.log" Aug 04 20:15:30.109229 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:30.109181 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/1.log" Aug 04 20:15:30.200031 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:30.199995 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd5c49894-ktdqh_c9efa644-f790-4e4b-8ddf-a9a75d1c9719/router/0.log" Aug 04 20:15:30.989324 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:30.989298 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bvf4l_65fb9b7e-8c2b-44f4-8a03-0ca043ee40a7/serve-healthcheck-canary/0.log" Aug 04 20:15:31.494361 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:31.494319 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-rd8q2_0092bf4b-84ef-4574-9726-4ed506b345c6/insights-operator/0.log" Aug 04 20:15:31.577376 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:31.577323 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbfzd_c5450ac4-a4d8-498e-96ab-6a6f6424d130/kube-rbac-proxy/0.log" Aug 04 20:15:31.597788 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:31.597754 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbfzd_c5450ac4-a4d8-498e-96ab-6a6f6424d130/exporter/0.log" Aug 04 20:15:31.618493 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:31.618468 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kbfzd_c5450ac4-a4d8-498e-96ab-6a6f6424d130/extractor/0.log" Aug 04 20:15:32.563810 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:32.563779 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bk9n8/perf-node-gather-daemonset-6csrc" Aug 04 20:15:35.030974 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:35.030941 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-ljzlw_1de709ab-960b-4bcc-a7b9-58a687572868/manager/0.log" Aug 04 20:15:35.047616 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:35.047588 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bb79d9bb-ljzlw_1de709ab-960b-4bcc-a7b9-58a687572868/copy-manifests/0.log" Aug 04 20:15:36.183545 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:36.183454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4_0f0bff41-45c7-4708-a703-b88f946d6da3/istio-proxy/0.log" Aug 04 20:15:36.250754 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:36.250721 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59d56c454c-45dps_532253aa-022c-4fff-930a-1125f88e260c/manager/0.log" Aug 04 20:15:36.553939 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:36.553895 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/0.log" Aug 04 20:15:36.565884 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:36.565853 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/1.log" Aug 04 20:15:36.622301 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:36.622261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd5c49894-ktdqh_c9efa644-f790-4e4b-8ddf-a9a75d1c9719/router/0.log" Aug 04 20:15:37.924323 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:37.924290 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmlvt4_0f0bff41-45c7-4708-a703-b88f946d6da3/istio-proxy/0.log" Aug 04 20:15:38.326616 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:38.326586 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/0.log" Aug 04 20:15:38.341862 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:38.341823 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-vpq2m_8a771bdd-68af-4d5c-9957-1164e4cfd1c4/istio-proxy/1.log" Aug 04 20:15:38.412520 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:38.412477 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dd5c49894-ktdqh_c9efa644-f790-4e4b-8ddf-a9a75d1c9719/router/0.log" Aug 04 20:15:41.092952 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:41.092925 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-mfqxh_64f740d6-37be-4d0d-87f9-ff984fe09397/migrator/0.log" Aug 04 20:15:41.130301 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:41.130270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-mfqxh_64f740d6-37be-4d0d-87f9-ff984fe09397/graceful-termination/0.log" Aug 04 20:15:41.480823 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:41.480716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-nlhhn_526fc512-89e6-4817-8d17-b561d2c1e7c9/kube-storage-version-migrator-operator/1.log" Aug 04 20:15:41.482096 ip-10-0-143-62 kubenswrapper[2579]: I0804 20:15:41.482071 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-nlhhn_526fc512-89e6-4817-8d17-b561d2c1e7c9/kube-storage-version-migrator-operator/0.log"