Jul 24 22:50:15.026153 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 22:50:15.026164 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 22:50:15.026171 ip-10-0-143-241 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 22:50:15.026384 ip-10-0-143-241 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 22:50:25.082953 ip-10-0-143-241 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 22:50:25.082970 ip-10-0-143-241 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 284093651bc0464b9a80bd280bdc8a7a -- Jul 24 22:52:20.606947 ip-10-0-143-241 systemd[1]: Starting Kubernetes Kubelet... Jul 24 22:52:21.011873 ip-10-0-143-241 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:52:21.011873 ip-10-0-143-241 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 22:52:21.011873 ip-10-0-143-241 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:52:21.011873 ip-10-0-143-241 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 22:52:21.011873 ip-10-0-143-241 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:52:21.012544 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.011947 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 22:52:21.016046 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016031 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:52:21.016046 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016046 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016050 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016054 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016057 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016060 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016062 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016065 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016068 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016071 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016073 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016076 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016078 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016086 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016089 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016094 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016098 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016102 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016106 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016109 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:52:21.016110 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016112 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016116 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016119 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016123 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016126 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016128 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016132 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016135 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016137 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016140 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016143 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016146 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016148 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016151 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016153 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016156 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016158 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016161 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016164 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016166 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:52:21.016555 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016169 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016172 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016174 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016177 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016179 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016182 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016185 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016187 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016190 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016193 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016196 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016198 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016201 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016203 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016206 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016209 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016212 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016214 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016217 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016219 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:52:21.017058 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016222 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016225 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016228 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016230 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016233 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016235 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016238 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016240 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016243 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016245 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016248 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016254 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016257 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016260 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016262 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016265 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016267 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016270 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016273 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016276 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:52:21.017536 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016279 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016281 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016284 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016287 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016289 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016292 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016295 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016298 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016302 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016306 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016309 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016312 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016315 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016317 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016320 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016322 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016325 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016328 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.016330 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:52:21.018033 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017721 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017729 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017732 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017735 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017739 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017742 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017746 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017749 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017752 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017754 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017757 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017760 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017763 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017765 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017768 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017771 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017774 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017776 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017779 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017782 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:52:21.018498 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017784 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017787 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017790 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017792 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017795 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017798 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017800 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017803 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017806 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017808 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017811 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017814 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017818 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017820 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017823 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017825 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017828 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017831 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017834 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:52:21.019029 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017836 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017838 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017842 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017844 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017846 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017849 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017851 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017854 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017857 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017860 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017863 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017866 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017868 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017872 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017875 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017878 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017881 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017883 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017886 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017888 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:52:21.019494 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017891 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017894 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017896 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017899 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017901 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017904 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017906 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017909 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017911 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017913 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017916 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017918 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017921 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017923 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017926 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017928 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017931 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017933 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017936 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:52:21.020017 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017938 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017941 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017944 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017946 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017951 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017955 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017958 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017961 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017964 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017966 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017969 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017972 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017974 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017977 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017980 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017982 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017984 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017987 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017989 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:52:21.020483 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017992 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.017995 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018345 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018350 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018353 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018356 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018359 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018362 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018365 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018367 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018372 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018375 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018378 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018381 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018385 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018388 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018391 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018394 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018397 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:52:21.020973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018400 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018403 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018405 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018408 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018410 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018413 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018415 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018418 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018421 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018423 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018426 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018430 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018433 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018436 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018438 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018441 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018443 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018446 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018448 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018451 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:52:21.021469 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018454 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018456 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018459 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018461 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018463 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018466 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018468 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018477 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018479 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018482 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018485 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018487 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018490 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018493 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018495 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018498 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018501 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018504 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018506 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018509 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:52:21.021953 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018512 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018514 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018517 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018520 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018522 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018525 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018528 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018530 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018533 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018535 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018539 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018541 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018544 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018547 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018550 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018552 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018554 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018557 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018560 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018564 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:52:21.022456 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018567 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018570 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018574 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018577 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018580 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018584 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018586 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018589 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018592 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018595 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018597 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018600 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018602 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018605 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018607 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018610 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018612 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018615 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018617 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:52:21.022986 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018620 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018623 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.018625 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018707 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018714 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018720 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018724 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018729 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018733 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018737 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018741 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018744 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018749 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018752 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018755 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018759 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018762 2562 flags.go:64] FLAG: --cgroup-root="" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018766 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018768 2562 flags.go:64] FLAG: --client-ca-file="" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018771 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018774 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018779 2562 flags.go:64] FLAG: --cluster-domain="" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018781 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018785 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 22:52:21.023468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018788 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018791 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018795 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018798 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018801 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018804 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018807 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018810 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018813 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018816 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018819 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018822 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018826 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018828 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018831 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018834 2562 flags.go:64] FLAG: --enable-server="true" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018837 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018842 2562 flags.go:64] FLAG: --event-burst="100" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018845 2562 flags.go:64] FLAG: --event-qps="50" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018848 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018852 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018856 2562 flags.go:64] FLAG: --eviction-hard="" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018862 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018865 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018868 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 22:52:21.024067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018871 2562 flags.go:64] FLAG: --eviction-soft="" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018874 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018877 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018880 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018882 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018885 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018888 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018891 2562 flags.go:64] FLAG: --feature-gates="" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018894 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018898 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018901 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018904 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018907 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018909 2562 flags.go:64] FLAG: --help="false" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018913 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018915 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018918 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018921 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018925 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018929 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018932 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018934 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018937 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 22:52:21.024664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018940 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018943 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018947 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018949 2562 flags.go:64] FLAG: --kube-reserved="" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018954 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018957 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018961 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018964 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018966 2562 flags.go:64] FLAG: --lock-file="" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018969 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018972 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018975 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018980 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018983 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018986 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018988 2562 flags.go:64] FLAG: --logging-format="text" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018991 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018995 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.018997 2562 flags.go:64] FLAG: --manifest-url="" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019000 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019004 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019007 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019011 2562 flags.go:64] FLAG: --max-pods="110" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019014 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019017 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 22:52:21.025267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019020 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019022 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019025 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019028 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019031 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019048 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019051 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019054 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019057 2562 flags.go:64] FLAG: --pod-cidr="" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019061 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019065 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019069 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019072 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019075 2562 flags.go:64] FLAG: --port="10250" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019080 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019083 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0375a887ef9e1cdf1" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019086 2562 flags.go:64] FLAG: --qos-reserved="" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019089 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019092 2562 flags.go:64] FLAG: --register-node="true" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019095 2562 flags.go:64] FLAG: --register-with-taints="" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019098 2562 flags.go:64] FLAG: --registry-burst="10" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019101 2562 flags.go:64] FLAG: --registry-qps="5" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019104 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019106 2562 flags.go:64] FLAG: --reserved-memory="" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019110 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 22:52:21.025903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019113 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019115 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019119 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019121 2562 flags.go:64] FLAG: --runonce="false" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019124 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019127 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019130 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019133 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019136 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019139 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019142 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019145 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019148 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019150 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019153 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019156 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019159 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019162 2562 flags.go:64] FLAG: --system-cgroups="" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019166 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019172 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019174 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019179 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019183 2562 flags.go:64] FLAG: --tls-min-version="" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019186 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019189 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 22:52:21.026556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019192 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019195 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019198 2562 flags.go:64] FLAG: --v="2" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019201 2562 flags.go:64] FLAG: --version="false" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019205 2562 flags.go:64] FLAG: --vmodule="" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019209 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.019212 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019330 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019335 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019339 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019342 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019345 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019348 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019350 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019353 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019356 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019358 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019361 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019365 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019368 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019372 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:52:21.027169 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019375 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019379 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019382 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019384 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019389 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019391 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019394 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019398 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019401 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019404 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019406 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019409 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019412 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019414 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019417 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019419 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019422 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019425 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019427 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:52:21.027995 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019430 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019432 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019435 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019438 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019440 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019443 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019445 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019448 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019452 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019456 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019459 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019462 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019465 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019467 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019470 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019473 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019475 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019479 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019482 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:52:21.028654 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019485 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019489 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019492 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019495 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019497 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019500 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019503 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019506 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019509 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019512 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019514 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019517 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019520 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019523 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019525 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019528 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019530 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019533 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019536 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019538 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:52:21.029248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019541 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019543 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019546 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019548 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019551 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019554 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019556 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019558 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019561 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019564 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019567 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019569 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019573 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019577 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019580 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019582 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019585 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019588 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019590 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019593 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:52:21.029761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019596 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019598 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019601 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019603 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019606 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019608 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.019611 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.020150 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.027600 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.027636 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.027654 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.027662 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027928 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027942 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027951 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:52:21.030239 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027959 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027967 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027975 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027982 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027990 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.027999 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028012 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028019 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028029 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028072 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028080 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028088 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028096 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028105 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028114 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028121 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028129 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028138 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028151 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028158 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:52:21.030615 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028166 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028172 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028179 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028186 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028221 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028227 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028232 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028237 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028241 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028245 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028249 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028259 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028264 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028268 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028272 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028276 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028280 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028285 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028289 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028292 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:52:21.031118 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028297 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028301 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028360 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028376 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028384 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028389 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028395 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028400 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028405 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028410 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028415 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028420 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028425 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028430 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028440 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028444 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028449 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028454 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028458 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:52:21.031603 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028463 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028467 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028611 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028626 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028630 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028633 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028636 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028639 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028643 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028646 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028649 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028653 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028656 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028659 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028663 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028666 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028669 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028686 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028691 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028694 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:52:21.032084 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028697 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028700 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028703 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028706 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028709 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028711 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028714 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028717 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028719 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028722 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028725 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028727 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028730 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028733 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028736 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028741 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:52:21.032582 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028744 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.028750 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028881 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028885 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028888 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028891 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028894 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028897 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028900 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028903 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028906 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028909 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028911 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028914 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028916 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:52:21.032980 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028919 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028922 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028924 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028927 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028929 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028932 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028934 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028937 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028939 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028943 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028947 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028950 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028953 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028956 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028958 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028961 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028964 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028968 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028971 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:52:21.033359 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028973 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028976 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028978 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028981 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028983 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028986 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028988 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028991 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028993 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028996 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.028998 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029001 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029004 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029006 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029009 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029011 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029014 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029016 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029019 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029021 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:52:21.033833 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029024 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029026 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029029 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029031 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029034 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029036 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029039 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029042 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029044 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029047 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029050 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029053 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029055 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029058 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029061 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029063 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029066 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029069 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029071 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029074 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:52:21.034309 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029076 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029080 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029084 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029087 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029090 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029094 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029097 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029101 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029103 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029106 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029109 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029111 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029114 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029116 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029119 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029122 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029124 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029126 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029129 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:52:21.034815 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029131 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029134 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029137 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029139 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029142 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029145 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029148 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:21.029150 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.029155 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.029284 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.031367 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.032634 2562 server.go:1019] "Starting client certificate rotation" Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.032702 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:52:21.035298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.032740 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:52:21.035624 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.035058 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:52:21.036995 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.036979 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:52:21.048160 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.048138 2562 log.go:25] "Validated CRI v1 runtime API" Jul 24 22:52:21.051373 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.051356 2562 log.go:25] "Validated CRI v1 image API" Jul 24 22:52:21.052161 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.052141 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 22:52:21.054447 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.054424 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 96f63be8-5e82-4d65-a631-acc4f490ad2f:/dev/nvme0n1p3 ebb3adad-c1d5-4a1d-ad7e-385a5ae1ddb5:/dev/nvme0n1p4] Jul 24 22:52:21.054518 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.054447 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 22:52:21.060855 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.060744 2562 manager.go:216] Machine: {Timestamp:2026-07-24 22:52:21.058552674 +0000 UTC m=+0.333521760 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098174 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254ac8d046fa76aad88c42e3d05e05 SystemUUID:ec254ac8-d046-fa76-aad8-8c42e3d05e05 BootID:28409365-1bc0-464b-9a80-bd280bdc8a7a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:50:e9:92:17:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:50:e9:92:17:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:d8:45:e0:3c:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 22:52:21.060855 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.060847 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 22:52:21.061001 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.060950 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 22:52:21.061125 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.061103 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:52:21.063975 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.063950 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 22:52:21.064535 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.063985 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 22:52:21.065647 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.065184 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 22:52:21.065729 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.065656 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 22:52:21.065729 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.065667 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:52:21.066616 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.066438 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:52:21.067391 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.067380 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 22:52:21.069057 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069045 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:52:21.069158 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069149 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 22:52:21.069196 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069178 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 24 22:52:21.069196 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069187 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 22:52:21.069261 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069200 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 22:52:21.069261 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069209 2562 kubelet.go:398] "Adding apiserver pod source" Jul 24 22:52:21.069261 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069218 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 22:52:21.069600 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069589 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:52:21.069638 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.069606 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:52:21.072025 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.072006 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 22:52:21.073838 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.073825 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 22:52:21.073878 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.073854 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 22:52:21.075950 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075938 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075956 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075962 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075967 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075976 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075982 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075989 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 22:52:21.075996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.075994 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 22:52:21.076179 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.076001 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 22:52:21.076179 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.076006 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 22:52:21.076179 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.076019 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 22:52:21.076179 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.076024 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 22:52:21.076848 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.076834 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 22:52:21.076880 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.076849 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 22:52:21.079822 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.079809 2562 server.go:1268] "Started kubelet" Jul 24 22:52:21.080031 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.079942 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 22:52:21.080125 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.080060 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 22:52:21.080643 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.080616 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 22:52:21.080662 ip-10-0-143-241 systemd[1]: Started Kubernetes Kubelet. Jul 24 22:52:21.081481 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.081245 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 22:52:21.081481 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.081362 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 22:52:21.081481 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.081395 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 22:52:21.081862 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.081719 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 22:52:21.083125 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.083108 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 24 22:52:21.083636 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.083619 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-48fvs" Jul 24 22:52:21.085478 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.085452 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 22:52:21.085557 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.084524 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-241.ec2.internal.18c55b993fbd9ae3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-241.ec2.internal,UID:ip-10-0-143-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-241.ec2.internal,},FirstTimestamp:2026-07-24 22:52:21.079784163 +0000 UTC m=+0.354753249,LastTimestamp:2026-07-24 22:52:21.079784163 +0000 UTC m=+0.354753249,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-241.ec2.internal,}" Jul 24 22:52:21.088472 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.088451 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 22:52:21.088573 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.088450 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 22:52:21.088627 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.088613 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.089060 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089043 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 22:52:21.089184 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089045 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 22:52:21.089283 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089187 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 22:52:21.089331 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089298 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 24 22:52:21.089331 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089307 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 24 22:52:21.089410 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089395 2562 factory.go:153] Registering CRI-O factory Jul 24 22:52:21.089454 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089410 2562 factory.go:223] Registration of the crio container factory successfully Jul 24 22:52:21.089500 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089470 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 22:52:21.089500 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089480 2562 factory.go:55] Registering systemd factory Jul 24 22:52:21.089500 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089487 2562 factory.go:223] Registration of the systemd container factory successfully Jul 24 22:52:21.089628 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089508 2562 factory.go:103] Registering Raw factory Jul 24 22:52:21.089628 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089521 2562 manager.go:1196] Started watching for new ooms in manager Jul 24 22:52:21.089935 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.089922 2562 manager.go:349] Starting recovery of all containers Jul 24 22:52:21.091200 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.091177 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-48fvs" Jul 24 22:52:21.099981 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.099960 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:21.101534 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.101460 2562 manager.go:354] Recovery completed Jul 24 22:52:21.105027 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.105004 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.105521 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.105509 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.108155 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108136 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.108233 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108167 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.108233 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108191 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.108714 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108657 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 22:52:21.108714 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108687 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.108801 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108725 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 22:52:21.108801 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.108747 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:52:21.109366 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.109353 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.109420 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.109377 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.109420 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.109386 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.110474 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.110459 2562 policy_none.go:49] "None policy: Start" Jul 24 22:52:21.110523 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.110482 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 22:52:21.110523 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.110496 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 22:52:21.111647 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.111634 2562 policy_none.go:47] "Start" Jul 24 22:52:21.149323 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149203 2562 manager.go:337] "Starting Device Plugin manager" Jul 24 22:52:21.149410 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.149343 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 22:52:21.149410 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149357 2562 server.go:87] "Starting device plugin registration server" Jul 24 22:52:21.149573 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149561 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 22:52:21.149604 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149575 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 22:52:21.149703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149670 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 22:52:21.149795 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149764 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 22:52:21.149795 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.149773 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 22:52:21.150244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.150219 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 22:52:21.150312 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.150273 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.250958 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.250933 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.251942 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.251889 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.251942 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.251929 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.252066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.251946 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.252066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.251984 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.261071 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.261054 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.261200 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.261080 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-241.ec2.internal\": node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.278673 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.278620 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.292412 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.292386 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 22:52:21.293603 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.293580 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 22:52:21.293679 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.293616 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 22:52:21.293679 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.293636 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 22:52:21.293679 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.293666 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 22:52:21.296297 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.296275 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:21.383139 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.383108 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.394353 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.394330 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal"] Jul 24 22:52:21.394422 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.394412 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.395329 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.395312 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.395391 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.395351 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.395391 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.395365 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.397211 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397198 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.397344 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397330 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.397402 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397360 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.397975 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397955 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.398042 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397986 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.398042 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397997 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.398120 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.397955 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.398120 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.398074 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.398120 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.398090 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.399299 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.399285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.399350 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.399319 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:52:21.400066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.400049 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:52:21.400145 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.400074 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:52:21.400145 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.400093 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:52:21.421480 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.421457 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.426279 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.426263 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-241.ec2.internal\" not found" node="ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.483500 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.483476 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.492039 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.492018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa302ee4376c6431a39ad601ac23daaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"aa302ee4376c6431a39ad601ac23daaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.492109 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.492045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa302ee4376c6431a39ad601ac23daaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"aa302ee4376c6431a39ad601ac23daaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.492109 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.492068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8c2bd3954b3c3942399264f92d6d2f7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"b8c2bd3954b3c3942399264f92d6d2f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.584415 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.584343 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.592237 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.592217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa302ee4376c6431a39ad601ac23daaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"aa302ee4376c6431a39ad601ac23daaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.592296 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.592265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa302ee4376c6431a39ad601ac23daaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"aa302ee4376c6431a39ad601ac23daaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.592340 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.592324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa302ee4376c6431a39ad601ac23daaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"aa302ee4376c6431a39ad601ac23daaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.592380 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.592353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8c2bd3954b3c3942399264f92d6d2f7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"b8c2bd3954b3c3942399264f92d6d2f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.592420 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.592382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8c2bd3954b3c3942399264f92d6d2f7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-241.ec2.internal\" (UID: \"b8c2bd3954b3c3942399264f92d6d2f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.592420 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.592400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa302ee4376c6431a39ad601ac23daaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal\" (UID: \"aa302ee4376c6431a39ad601ac23daaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.692035 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.692010 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.724377 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.724345 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.729841 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.729823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" Jul 24 22:52:21.792309 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.792281 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.894372 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:21.894316 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-241.ec2.internal\" not found" Jul 24 22:52:21.968926 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.968901 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:21.989424 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:21.989396 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" Jul 24 22:52:22.001137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.001118 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:52:22.003143 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.003130 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" Jul 24 22:52:22.009487 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.009471 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:52:22.031951 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.031918 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 22:52:22.032353 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.032036 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 22:52:22.032353 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.032070 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 22:52:22.032353 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.032081 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 22:52:22.069561 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.069538 2562 apiserver.go:52] "Watching apiserver" Jul 24 22:52:22.070655 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.070638 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:22.078340 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.078318 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 22:52:22.079534 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.079513 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-d7wkj","kube-system/global-pull-secret-syncer-g75zv","kube-system/konnectivity-agent-fz57w","openshift-dns/node-resolver-q6dnt","openshift-image-registry/node-ca-24kn8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal","openshift-multus/multus-g48px","openshift-multus/network-metrics-daemon-c9rmb","openshift-network-operator/iptables-alerter-cbl8q","kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw","openshift-cluster-node-tuning-operator/tuned-89hrd","openshift-multus/multus-additional-cni-plugins-rdqk9","openshift-network-diagnostics/network-check-target-jjhgr"] Jul 24 22:52:22.081077 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.081062 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g48px" Jul 24 22:52:22.082337 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.082313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.083523 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.083504 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.083877 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.083861 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 22:52:22.084168 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.084130 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 22:52:22.084310 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.084296 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.084448 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.084433 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.084694 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.084676 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6nv4h\"" Jul 24 22:52:22.084790 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.084677 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.085720 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.085704 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 22:52:22.085816 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.085725 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-crqj6\"" Jul 24 22:52:22.085816 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.085742 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 22:52:22.085816 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.085784 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 22:52:22.085979 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.085816 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8wdlh\"" Jul 24 22:52:22.086034 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.086022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.086419 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.086401 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 22:52:22.087208 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.087190 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:22.087296 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.087259 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:22.087361 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.087338 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jpz2p\"" Jul 24 22:52:22.087417 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.087379 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.088144 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.088119 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.088443 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.088428 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.088495 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.088478 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.088624 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.088606 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 22:52:22.088780 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.088759 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wqvwk\"" Jul 24 22:52:22.089034 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.089016 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 22:52:22.089088 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.089019 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.089834 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.089815 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.091045 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.091026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.091186 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.091077 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:22.091260 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.091182 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.091321 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.091305 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qvd4w\"" Jul 24 22:52:22.091370 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.091349 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 22:52:22.091370 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.091356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.092186 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.092160 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 22:52:22.092271 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.092239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.092359 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.092306 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:22.092863 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.092847 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t868z\"" Jul 24 22:52:22.093061 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.093042 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.093144 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.093042 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.093144 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.093044 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 22:52:22.093144 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.093136 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 22:52:22.093589 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.093571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.094123 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.094107 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 22:52:22.094800 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.094784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.094947 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.094930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-ovn\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095024 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.094958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095024 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-ovnkube-config\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095130 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-env-overrides\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095130 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glqj\" (UniqueName: \"kubernetes.io/projected/5dad8c34-9078-4e76-8f4c-5b1590840294-kube-api-access-7glqj\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095130 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-serviceca\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.095288 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-system-cni-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095288 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-cnibin\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095288 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:22.095288 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-kubelet\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095288 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-systemd-units\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-var-lib-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-cni-netd\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-run-netns\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-systemd\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-node-log\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-run-ovn-kubernetes\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-slash\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095513 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5dad8c34-9078-4e76-8f4c-5b1590840294-ovn-node-metrics-cert\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-etc-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-ovnkube-script-lib\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0-konnectivity-ca\") pod \"konnectivity-agent-fz57w\" (UID: \"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0\") " pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmw7\" (UniqueName: \"kubernetes.io/projected/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-kube-api-access-slmw7\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-multus-certs\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsppp\" (UniqueName: \"kubernetes.io/projected/57c0de4c-56cd-4957-9ae9-b95c62191c30-kube-api-access-hsppp\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-host-slash\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-host\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-k8s-cni-cncf-io\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-os-release\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-hostroot\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-conf-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.095893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-cni-bin\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095918 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-cni-bin\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.095974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-etc-kubernetes\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-cnibin\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096082 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntsh\" (UniqueName: \"kubernetes.io/projected/ce701a92-c7b7-43bf-8661-d2d5078f4836-kube-api-access-7ntsh\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-log-socket\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096156 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-kubelet\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096191 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbst\" (UniqueName: \"kubernetes.io/projected/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-kube-api-access-mzbst\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-socket-dir-parent\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-cni-multus\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096302 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mtcd\"" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096319 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 22:47:21 +0000 UTC" deadline="2028-01-07 18:58:05.860287696 +0000 UTC" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096339 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12764h5m43.763950758s" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-system-cni-dir\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096273 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.096550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n87w7\" (UniqueName: \"kubernetes.io/projected/e0caf250-1643-481c-a696-e2e6dfe23b61-kube-api-access-n87w7\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096479 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce701a92-c7b7-43bf-8661-d2d5078f4836-tmp-dir\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-cni-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57c0de4c-56cd-4957-9ae9-b95c62191c30-cni-binary-copy\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-daemon-config\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce701a92-c7b7-43bf-8661-d2d5078f4836-hosts-file\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-iptables-alerter-script\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.096702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0-agent-certs\") pod \"konnectivity-agent-fz57w\" (UID: \"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0\") " pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.097143 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.097231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-os-release\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.097268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-netns\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.097902 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.097306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.098572 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.098039 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 22:52:22.098572 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.098099 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q497r\"" Jul 24 22:52:22.098572 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.098447 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:52:22.103102 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.103062 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:52:22.124593 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.124575 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hlp6f" Jul 24 22:52:22.133019 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.133000 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hlp6f" Jul 24 22:52:22.189906 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.189878 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 22:52:22.197738 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-etc-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.197854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-ovnkube-script-lib\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.197854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0-konnectivity-ca\") pod \"konnectivity-agent-fz57w\" (UID: \"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0\") " pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.197854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-etc-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.197854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmw7\" (UniqueName: \"kubernetes.io/projected/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-kube-api-access-slmw7\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.197854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-multus-certs\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsppp\" (UniqueName: \"kubernetes.io/projected/57c0de4c-56cd-4957-9ae9-b95c62191c30-kube-api-access-hsppp\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-socket-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-host-slash\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-host\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.197986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-k8s-cni-cncf-io\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysconfig\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysctl-d\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-dbus\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.198137 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9vt\" (UniqueName: \"kubernetes.io/projected/54cb923e-1fe5-4cef-a83a-50676bf4995b-kube-api-access-fs9vt\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-os-release\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-hostroot\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-conf-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysctl-conf\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-cni-bin\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-cni-bin\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-etc-kubernetes\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-cnibin\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198378 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntsh\" (UniqueName: \"kubernetes.io/projected/ce701a92-c7b7-43bf-8661-d2d5078f4836-kube-api-access-7ntsh\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-multus-certs\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-conf-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-log-socket\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-k8s-cni-cncf-io\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-os-release\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-host-slash\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-etc-kubernetes\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.198685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-hostroot\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198670 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-host\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198674 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-cnibin\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-cni-bin\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-cni-bin\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-log-socket\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-kubelet\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-ovnkube-script-lib\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-systemd\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-device-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-kubelet\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbst\" (UniqueName: \"kubernetes.io/projected/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-kube-api-access-mzbst\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198918 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-run\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0-konnectivity-ca\") pod \"konnectivity-agent-fz57w\" (UID: \"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0\") " pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-socket-dir-parent\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.199703 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-cni-multus\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-modprobe-d\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.198865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-var-lib-cni-multus\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-socket-dir-parent\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-system-cni-dir\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199192 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0caf250-1643-481c-a696-e2e6dfe23b61-system-cni-dir\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n87w7\" (UniqueName: \"kubernetes.io/projected/e0caf250-1643-481c-a696-e2e6dfe23b61-kube-api-access-n87w7\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce701a92-c7b7-43bf-8661-d2d5078f4836-tmp-dir\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-cni-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57c0de4c-56cd-4957-9ae9-b95c62191c30-cni-binary-copy\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-kubernetes\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-daemon-config\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.200491 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-host\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-sys-fs\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce701a92-c7b7-43bf-8661-d2d5078f4836-hosts-file\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199802 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-cni-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-iptables-alerter-script\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce701a92-c7b7-43bf-8661-d2d5078f4836-hosts-file\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0-agent-certs\") pod \"konnectivity-agent-fz57w\" (UID: \"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0\") " pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-os-release\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-netns\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.199997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-sys\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccrd\" (UniqueName: \"kubernetes.io/projected/002ed68c-3e66-417f-9f2b-1b999ef83226-kube-api-access-7ccrd\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-tmp\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-registration-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-ovn\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.201405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57c0de4c-56cd-4957-9ae9-b95c62191c30-cni-binary-copy\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-ovnkube-config\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-env-overrides\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200285 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glqj\" (UniqueName: \"kubernetes.io/projected/5dad8c34-9078-4e76-8f4c-5b1590840294-kube-api-access-7glqj\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57c0de4c-56cd-4957-9ae9-b95c62191c30-multus-daemon-config\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-serviceca\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-system-cni-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-cnibin\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-lib-modules\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200467 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-iptables-alerter-script\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200479 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-host-run-netns\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-ovn\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-kubelet\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-systemd-units\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.202251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-var-lib-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-cni-netd\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbk5g\" (UniqueName: \"kubernetes.io/projected/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-kube-api-access-bbk5g\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200832 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-ovnkube-config\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-kubelet\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-run-netns\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-system-cni-dir\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-systemd-units\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-var-lib-openvswitch\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-systemd\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200971 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-cni-netd\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-cnibin\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.200972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-node-log\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-run-ovn-kubernetes\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-node-log\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-var-lib-kubelet\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.203152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-kubelet-config\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57c0de4c-56cd-4957-9ae9-b95c62191c30-os-release\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0caf250-1643-481c-a696-e2e6dfe23b61-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-run-systemd\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-slash\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5dad8c34-9078-4e76-8f4c-5b1590840294-env-overrides\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5dad8c34-9078-4e76-8f4c-5b1590840294-ovn-node-metrics-cert\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201255 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-run-ovn-kubernetes\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-tuned\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-slash\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5dad8c34-9078-4e76-8f4c-5b1590840294-host-run-netns\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201490 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-serviceca\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.203806 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.201893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce701a92-c7b7-43bf-8661-d2d5078f4836-tmp-dir\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.205160 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.205133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5dad8c34-9078-4e76-8f4c-5b1590840294-ovn-node-metrics-cert\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.205349 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.205328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0-agent-certs\") pod \"konnectivity-agent-fz57w\" (UID: \"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0\") " pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.207518 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.207228 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:22.207518 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.207258 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:22.207518 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.207273 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:22.207518 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.207340 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:22.707317326 +0000 UTC m=+1.982286401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:22.207518 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.207502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmw7\" (UniqueName: \"kubernetes.io/projected/a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6-kube-api-access-slmw7\") pod \"node-ca-24kn8\" (UID: \"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6\") " pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.208228 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.208197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntsh\" (UniqueName: \"kubernetes.io/projected/ce701a92-c7b7-43bf-8661-d2d5078f4836-kube-api-access-7ntsh\") pod \"node-resolver-q6dnt\" (UID: \"ce701a92-c7b7-43bf-8661-d2d5078f4836\") " pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.208652 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.208632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbst\" (UniqueName: \"kubernetes.io/projected/ff9e94c0-2b41-4a4c-9722-a5c65dbf688d-kube-api-access-mzbst\") pod \"iptables-alerter-cbl8q\" (UID: \"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d\") " pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.208772 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.208689 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n87w7\" (UniqueName: \"kubernetes.io/projected/e0caf250-1643-481c-a696-e2e6dfe23b61-kube-api-access-n87w7\") pod \"multus-additional-cni-plugins-rdqk9\" (UID: \"e0caf250-1643-481c-a696-e2e6dfe23b61\") " pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.208834 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.208812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsppp\" (UniqueName: \"kubernetes.io/projected/57c0de4c-56cd-4957-9ae9-b95c62191c30-kube-api-access-hsppp\") pod \"multus-g48px\" (UID: \"57c0de4c-56cd-4957-9ae9-b95c62191c30\") " pod="openshift-multus/multus-g48px" Jul 24 22:52:22.209767 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.209727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glqj\" (UniqueName: \"kubernetes.io/projected/5dad8c34-9078-4e76-8f4c-5b1590840294-kube-api-access-7glqj\") pod \"ovnkube-node-d7wkj\" (UID: \"5dad8c34-9078-4e76-8f4c-5b1590840294\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.259921 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.259893 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa302ee4376c6431a39ad601ac23daaa.slice/crio-75b5222e57e8e1b78750fbe4f2ddff964acc59be7922b816da45a77e72fe7345 WatchSource:0}: Error finding container 75b5222e57e8e1b78750fbe4f2ddff964acc59be7922b816da45a77e72fe7345: Status 404 returned error can't find the container with id 75b5222e57e8e1b78750fbe4f2ddff964acc59be7922b816da45a77e72fe7345 Jul 24 22:52:22.260403 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.260381 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c2bd3954b3c3942399264f92d6d2f7.slice/crio-29a9052a53056715a63a73c3fd8c3dc015fffd08b29b7d66d478669a9e3e5457 WatchSource:0}: Error finding container 29a9052a53056715a63a73c3fd8c3dc015fffd08b29b7d66d478669a9e3e5457: Status 404 returned error can't find the container with id 29a9052a53056715a63a73c3fd8c3dc015fffd08b29b7d66d478669a9e3e5457 Jul 24 22:52:22.264093 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.264076 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:52:22.297175 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.297132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"aa302ee4376c6431a39ad601ac23daaa","Type":"ContainerStarted","Data":"75b5222e57e8e1b78750fbe4f2ddff964acc59be7922b816da45a77e72fe7345"} Jul 24 22:52:22.298290 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.298267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" event={"ID":"b8c2bd3954b3c3942399264f92d6d2f7","Type":"ContainerStarted","Data":"29a9052a53056715a63a73c3fd8c3dc015fffd08b29b7d66d478669a9e3e5457"} Jul 24 22:52:22.301563 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-sys\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.301664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccrd\" (UniqueName: \"kubernetes.io/projected/002ed68c-3e66-417f-9f2b-1b999ef83226-kube-api-access-7ccrd\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.301664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-tmp\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.301664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-registration-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.301854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-sys\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.301854 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-lib-modules\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.301958 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.301958 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbk5g\" (UniqueName: \"kubernetes.io/projected/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-kube-api-access-bbk5g\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.301958 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-registration-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.301958 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-var-lib-kubelet\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-kubelet-config\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.301998 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.301998 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-lib-modules\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-tuned\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-var-lib-kubelet\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.302054 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:52:22.80203509 +0000 UTC m=+2.077004179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-kubelet-config\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-socket-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysconfig\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysctl-d\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302173 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-dbus\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-dbus\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9vt\" (UniqueName: \"kubernetes.io/projected/54cb923e-1fe5-4cef-a83a-50676bf4995b-kube-api-access-fs9vt\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysconfig\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysctl-conf\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-systemd\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-device-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysctl-d\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-socket-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-run\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-systemd\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-run\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-device-dir\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-sysctl-conf\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.302726 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-modprobe-d\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.302535 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-kubernetes\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.302582 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:22.802565038 +0000 UTC m=+2.077534118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-modprobe-d\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-kubernetes\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-host\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-sys-fs\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54cb923e-1fe5-4cef-a83a-50676bf4995b-sys-fs\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.303657 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.302722 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-host\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.304267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.304180 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-etc-tuned\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.304487 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.304471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-tmp\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.313031 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.313011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbk5g\" (UniqueName: \"kubernetes.io/projected/fba7d161-4bae-4bb8-b6f3-2eeaeecb0077-kube-api-access-bbk5g\") pod \"tuned-89hrd\" (UID: \"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077\") " pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.313331 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.313311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccrd\" (UniqueName: \"kubernetes.io/projected/002ed68c-3e66-417f-9f2b-1b999ef83226-kube-api-access-7ccrd\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.316830 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.316803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9vt\" (UniqueName: \"kubernetes.io/projected/54cb923e-1fe5-4cef-a83a-50676bf4995b-kube-api-access-fs9vt\") pod \"aws-ebs-csi-driver-node-9gsmw\" (UID: \"54cb923e-1fe5-4cef-a83a-50676bf4995b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.413887 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.413829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g48px" Jul 24 22:52:22.421537 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.421516 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c0de4c_56cd_4957_9ae9_b95c62191c30.slice/crio-5a2a37c59c312798617388cae2767de21c5fac9e30f967447a2b26875bd19378 WatchSource:0}: Error finding container 5a2a37c59c312798617388cae2767de21c5fac9e30f967447a2b26875bd19378: Status 404 returned error can't find the container with id 5a2a37c59c312798617388cae2767de21c5fac9e30f967447a2b26875bd19378 Jul 24 22:52:22.431138 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.431121 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" Jul 24 22:52:22.438161 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.438128 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0caf250_1643_481c_a696_e2e6dfe23b61.slice/crio-9983da50b16672d7b6e1f26e495fb8874dc8c294c319b6dab5c5549c6856ce4a WatchSource:0}: Error finding container 9983da50b16672d7b6e1f26e495fb8874dc8c294c319b6dab5c5549c6856ce4a: Status 404 returned error can't find the container with id 9983da50b16672d7b6e1f26e495fb8874dc8c294c319b6dab5c5549c6856ce4a Jul 24 22:52:22.438873 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.438853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:22.447260 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.447234 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be5c3d1_c3f6_4834_9a39_e5091bd2f5c0.slice/crio-fa28ac6a91a1c47264313e07048005eff7a361c0695816a72b8216da725a83c9 WatchSource:0}: Error finding container fa28ac6a91a1c47264313e07048005eff7a361c0695816a72b8216da725a83c9: Status 404 returned error can't find the container with id fa28ac6a91a1c47264313e07048005eff7a361c0695816a72b8216da725a83c9 Jul 24 22:52:22.462208 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.462187 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6dnt" Jul 24 22:52:22.467107 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.467084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-24kn8" Jul 24 22:52:22.468993 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.468969 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce701a92_c7b7_43bf_8661_d2d5078f4836.slice/crio-5ff11055910f1b7d2db6d0bb5114660e68a33ccfb3721b07bbf2db2cd4854c7e WatchSource:0}: Error finding container 5ff11055910f1b7d2db6d0bb5114660e68a33ccfb3721b07bbf2db2cd4854c7e: Status 404 returned error can't find the container with id 5ff11055910f1b7d2db6d0bb5114660e68a33ccfb3721b07bbf2db2cd4854c7e Jul 24 22:52:22.473877 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.473854 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69aefd5_23e0_45b8_b9d0_a87a4e2aabe6.slice/crio-97ac53d6507bec6a4bb0c46b5d918d0ff04e46afdb19447db6d9ccb8c738cc18 WatchSource:0}: Error finding container 97ac53d6507bec6a4bb0c46b5d918d0ff04e46afdb19447db6d9ccb8c738cc18: Status 404 returned error can't find the container with id 97ac53d6507bec6a4bb0c46b5d918d0ff04e46afdb19447db6d9ccb8c738cc18 Jul 24 22:52:22.483027 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.483009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cbl8q" Jul 24 22:52:22.491090 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.491063 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9e94c0_2b41_4a4c_9722_a5c65dbf688d.slice/crio-00841a3b08acd7dda3f98d0e3b5f2fc27aa861959ea45c4e1820abe4d53be12d WatchSource:0}: Error finding container 00841a3b08acd7dda3f98d0e3b5f2fc27aa861959ea45c4e1820abe4d53be12d: Status 404 returned error can't find the container with id 00841a3b08acd7dda3f98d0e3b5f2fc27aa861959ea45c4e1820abe4d53be12d Jul 24 22:52:22.505803 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.505784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:22.512527 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.512504 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dad8c34_9078_4e76_8f4c_5b1590840294.slice/crio-e98518ef716fe093783b754a976062589026bb5853d94fb846a876e77963028c WatchSource:0}: Error finding container e98518ef716fe093783b754a976062589026bb5853d94fb846a876e77963028c: Status 404 returned error can't find the container with id e98518ef716fe093783b754a976062589026bb5853d94fb846a876e77963028c Jul 24 22:52:22.537355 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.537337 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" Jul 24 22:52:22.542814 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.542792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89hrd" Jul 24 22:52:22.544692 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.544668 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cb923e_1fe5_4cef_a83a_50676bf4995b.slice/crio-0dfd8605d503299d2fed5dc3011e46f59fb54c6d9056aadfca3c8e6b5e583b2e WatchSource:0}: Error finding container 0dfd8605d503299d2fed5dc3011e46f59fb54c6d9056aadfca3c8e6b5e583b2e: Status 404 returned error can't find the container with id 0dfd8605d503299d2fed5dc3011e46f59fb54c6d9056aadfca3c8e6b5e583b2e Jul 24 22:52:22.549810 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:22.549788 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba7d161_4bae_4bb8_b6f3_2eeaeecb0077.slice/crio-aa3ecc6d2303d991967a9ef883aae7e4f1d642eb5eaed390b936decc56680f87 WatchSource:0}: Error finding container aa3ecc6d2303d991967a9ef883aae7e4f1d642eb5eaed390b936decc56680f87: Status 404 returned error can't find the container with id aa3ecc6d2303d991967a9ef883aae7e4f1d642eb5eaed390b936decc56680f87 Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.807132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.807209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:22.807239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807329 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807354 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807363 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807374 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807388 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807406 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:23.807383561 +0000 UTC m=+3.082352655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807426 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:23.8074146 +0000 UTC m=+3.082383675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:22.807483 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:22.807461 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:52:23.807434814 +0000 UTC m=+3.082403896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:23.134685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.134554 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:47:22 +0000 UTC" deadline="2028-04-13 22:57:41.871441238 +0000 UTC" Jul 24 22:52:23.134685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.134620 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15096h5m18.736828932s" Jul 24 22:52:23.294148 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.294114 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:23.294332 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.294243 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:23.294868 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.294842 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:23.294990 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.294957 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:23.319030 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.318978 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-24kn8" event={"ID":"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6","Type":"ContainerStarted","Data":"97ac53d6507bec6a4bb0c46b5d918d0ff04e46afdb19447db6d9ccb8c738cc18"} Jul 24 22:52:23.326506 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.326436 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6dnt" event={"ID":"ce701a92-c7b7-43bf-8661-d2d5078f4836","Type":"ContainerStarted","Data":"5ff11055910f1b7d2db6d0bb5114660e68a33ccfb3721b07bbf2db2cd4854c7e"} Jul 24 22:52:23.349923 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.349888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fz57w" event={"ID":"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0","Type":"ContainerStarted","Data":"fa28ac6a91a1c47264313e07048005eff7a361c0695816a72b8216da725a83c9"} Jul 24 22:52:23.361560 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.361529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerStarted","Data":"9983da50b16672d7b6e1f26e495fb8874dc8c294c319b6dab5c5549c6856ce4a"} Jul 24 22:52:23.382446 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.382416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89hrd" event={"ID":"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077","Type":"ContainerStarted","Data":"aa3ecc6d2303d991967a9ef883aae7e4f1d642eb5eaed390b936decc56680f87"} Jul 24 22:52:23.387426 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.387367 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:23.396166 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.396138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"e98518ef716fe093783b754a976062589026bb5853d94fb846a876e77963028c"} Jul 24 22:52:23.405857 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.405829 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cbl8q" event={"ID":"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d","Type":"ContainerStarted","Data":"00841a3b08acd7dda3f98d0e3b5f2fc27aa861959ea45c4e1820abe4d53be12d"} Jul 24 22:52:23.412487 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.412304 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g48px" event={"ID":"57c0de4c-56cd-4957-9ae9-b95c62191c30","Type":"ContainerStarted","Data":"5a2a37c59c312798617388cae2767de21c5fac9e30f967447a2b26875bd19378"} Jul 24 22:52:23.432284 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.432101 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" event={"ID":"54cb923e-1fe5-4cef-a83a-50676bf4995b","Type":"ContainerStarted","Data":"0dfd8605d503299d2fed5dc3011e46f59fb54c6d9056aadfca3c8e6b5e583b2e"} Jul 24 22:52:23.613027 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.612994 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:23.815727 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.815619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:23.815727 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.815693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:23.815958 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.815739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:23.815958 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.815868 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:23.815958 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.815932 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:52:25.815911646 +0000 UTC m=+5.090880721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:23.816346 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.816328 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:23.816424 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.816379 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:25.816363827 +0000 UTC m=+5.091332907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:23.816486 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.816462 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:23.816486 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.816481 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:23.816599 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.816494 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:23.816599 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:23.816530 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:25.816518953 +0000 UTC m=+5.091488032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:23.933053 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:23.933023 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:52:24.134959 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:24.134916 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:47:22 +0000 UTC" deadline="2028-02-19 02:12:06.980509096 +0000 UTC" Jul 24 22:52:24.134959 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:24.134955 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13779h19m42.845559074s" Jul 24 22:52:24.294303 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:24.294273 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:24.294508 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:24.294408 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:25.305968 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:25.305266 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:25.305968 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.305414 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:25.305968 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:25.305807 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:25.305968 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.305906 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:25.839165 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:25.839114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:25.839358 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:25.839200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:25.839358 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:25.839231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:25.839482 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839359 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:25.839482 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839424 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:52:29.839405926 +0000 UTC m=+9.114375006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:25.839598 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839514 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:25.839598 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839548 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:29.839536519 +0000 UTC m=+9.114505597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:25.839711 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839632 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:25.839711 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839651 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:25.839711 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839664 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:25.839711 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:25.839698 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:29.839687 +0000 UTC m=+9.114656076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:26.294145 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:26.294050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:26.294307 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:26.294208 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:27.294443 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:27.294411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:27.294963 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:27.294533 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:27.294963 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:27.294732 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:27.294963 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:27.294858 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:28.294598 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:28.294557 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:28.295099 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:28.294748 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:29.295434 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:29.294716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:29.295434 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.294864 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:29.295434 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:29.295262 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:29.295434 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.295346 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:29.874577 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:29.874543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:29.874781 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:29.874603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:29.874781 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:29.874636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:29.874781 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874765 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:29.874953 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874788 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:29.874953 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874812 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:29.874953 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874823 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:52:37.87480314 +0000 UTC m=+17.149772223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:29.874953 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874825 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:29.874953 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874880 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:37.87486408 +0000 UTC m=+17.149833169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:29.874953 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874934 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:29.875283 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:29.874975 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:37.874962129 +0000 UTC m=+17.149931205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:30.294851 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:30.294716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:30.294992 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:30.294886 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:31.293948 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:31.293915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:31.294413 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:31.294048 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:31.295863 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:31.295652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:31.295863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:31.295822 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:32.294531 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:32.294177 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:32.294531 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:32.294310 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:33.294941 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:33.294904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:33.295335 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:33.295015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:33.295335 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:33.295078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:33.295335 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:33.295199 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:34.294258 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:34.294225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:34.294434 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:34.294339 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:35.294914 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:35.294883 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:35.295347 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:35.294883 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:35.295347 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:35.294995 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:35.295347 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:35.295072 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:36.294314 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:36.294271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:36.294484 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:36.294412 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:37.293880 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:37.293842 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:37.294264 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.293990 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:37.294264 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:37.293849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:37.294264 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.294175 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:37.936610 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:37.936580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:37.936610 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:37.936614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:37.936647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936748 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936750 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936791 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:53.936777687 +0000 UTC m=+33.211746760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936818 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:52:53.936800436 +0000 UTC m=+33.211769523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936751 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936842 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936856 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:37.936894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:37.936899 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:53.93688713 +0000 UTC m=+33.211856205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:38.294002 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:38.293972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:38.294411 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:38.294104 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:39.294414 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:39.294379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:39.294907 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:39.294509 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:39.294907 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:39.294569 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:39.294907 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:39.294703 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:40.293973 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:40.293932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:40.294165 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:40.294055 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:41.297004 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:41.296972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:41.297443 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:41.297115 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:41.297443 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:41.297161 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:41.297443 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:41.297248 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:42.293912 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:42.293773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:42.294013 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:42.293984 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:42.486856 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:42.486828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" event={"ID":"b8c2bd3954b3c3942399264f92d6d2f7","Type":"ContainerStarted","Data":"0e74a4aa01502fef41f51fc6d89487aedfecb630bc0e78515569830b8f1af4a3"} Jul 24 22:52:43.297182 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.297074 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:43.297279 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.297141 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:43.297342 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:43.297248 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:43.297396 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:43.297373 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:43.491680 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.491451 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:52:43.492324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.492013 2562 generic.go:356] "Generic (PLEG): container finished" podID="5dad8c34-9078-4e76-8f4c-5b1590840294" containerID="95c29723973c9227c11a9d0aad66daec4da734b5e47bdc86aa1ff0ef68072a68" exitCode=1 Jul 24 22:52:43.492324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.492083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"e559f5075ea16b84c63699fe2297ff34896eba5ac605fb6d1796280e72129620"} Jul 24 22:52:43.492324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.492156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"7c9b482795fbe28d8bd0256e3ae38cb40bb30391cfc20e9aa09547ec8c089903"} Jul 24 22:52:43.492324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.492199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"b9c13e115cff140233159792e5ac2b79721a04b29a6bddb97b44d5f21b22ffae"} Jul 24 22:52:43.492324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.492229 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerDied","Data":"95c29723973c9227c11a9d0aad66daec4da734b5e47bdc86aa1ff0ef68072a68"} Jul 24 22:52:43.492324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.492267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"982b7ea7dc3ec6106ea9eec543016d780cb0af3c52480d7114a1957c0db25333"} Jul 24 22:52:43.493417 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.493399 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g48px" event={"ID":"57c0de4c-56cd-4957-9ae9-b95c62191c30","Type":"ContainerStarted","Data":"3c814d7546fcf09683b50f3d1c985780b67162721b65583cb513f55d62c01668"} Jul 24 22:52:43.494744 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.494719 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" event={"ID":"54cb923e-1fe5-4cef-a83a-50676bf4995b","Type":"ContainerStarted","Data":"2443758b17cdd09c61cb78a3dae7643e9eb6f61c91b131c1770bed0aafc883aa"} Jul 24 22:52:43.495912 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.495879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-24kn8" event={"ID":"a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6","Type":"ContainerStarted","Data":"ed75b69f62dae55790fa7cf91d8cec690a69282603c86d8db8d93016ecece4e3"} Jul 24 22:52:43.497174 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.497157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6dnt" event={"ID":"ce701a92-c7b7-43bf-8661-d2d5078f4836","Type":"ContainerStarted","Data":"e37766fbd1a42c56f5b1be66bdff5e9a1d3fb026957d04b912b360e81113bfb1"} Jul 24 22:52:43.498347 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.498330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fz57w" event={"ID":"8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0","Type":"ContainerStarted","Data":"d7684835175a9325b696a1cd62371d579b3c0850778510a6f7d2f3247c55ef3b"} Jul 24 22:52:43.499601 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.499578 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0caf250-1643-481c-a696-e2e6dfe23b61" containerID="5427ec7d96e2f04da7ab4f063d941349bcad3e8076cd5bbe2ac2949477165c97" exitCode=0 Jul 24 22:52:43.499712 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.499635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerDied","Data":"5427ec7d96e2f04da7ab4f063d941349bcad3e8076cd5bbe2ac2949477165c97"} Jul 24 22:52:43.500872 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.500853 2562 generic.go:356] "Generic (PLEG): container finished" podID="aa302ee4376c6431a39ad601ac23daaa" containerID="5a86a5a14faf2448e3dffd275362f6a613881101423723a9930fef29d32b21ef" exitCode=0 Jul 24 22:52:43.500960 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.500916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"aa302ee4376c6431a39ad601ac23daaa","Type":"ContainerDied","Data":"5a86a5a14faf2448e3dffd275362f6a613881101423723a9930fef29d32b21ef"} Jul 24 22:52:43.502196 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.502170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89hrd" event={"ID":"fba7d161-4bae-4bb8-b6f3-2eeaeecb0077","Type":"ContainerStarted","Data":"ca96db885e6c63ff815cbdfa75366b806e5ccde8a64ec4b95c17ee4d88169ed8"} Jul 24 22:52:43.508915 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.508868 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-241.ec2.internal" podStartSLOduration=21.508854087 podStartE2EDuration="21.508854087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:42.500494827 +0000 UTC m=+21.775463922" watchObservedRunningTime="2026-07-24 22:52:43.508854087 +0000 UTC m=+22.783823183" Jul 24 22:52:43.509687 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.509642 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g48px" podStartSLOduration=2.48944968 podStartE2EDuration="22.509631126s" totalImagesPullingTime="20.020181446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:43.509031075 +0000 UTC m=+22.784000162" watchObservedRunningTime="2026-07-24 22:52:43.509631126 +0000 UTC m=+22.784600220" Jul 24 22:52:43.525713 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.525642 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-89hrd" podStartSLOduration=2.866843231 podStartE2EDuration="22.525625146s" totalImagesPullingTime="19.658781915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:43.525599509 +0000 UTC m=+22.800568606" watchObservedRunningTime="2026-07-24 22:52:43.525625146 +0000 UTC m=+22.800594245" Jul 24 22:52:43.581066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.581015 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fz57w" podStartSLOduration=2.821422587 podStartE2EDuration="22.580996427s" totalImagesPullingTime="19.75957384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:43.563233684 +0000 UTC m=+22.838202780" watchObservedRunningTime="2026-07-24 22:52:43.580996427 +0000 UTC m=+22.855965525" Jul 24 22:52:43.581196 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.581119 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q6dnt" podStartSLOduration=2.837076658 podStartE2EDuration="22.581114587s" totalImagesPullingTime="19.744037929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:43.581076523 +0000 UTC m=+22.856045617" watchObservedRunningTime="2026-07-24 22:52:43.581114587 +0000 UTC m=+22.856083685" Jul 24 22:52:43.595398 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:43.595294 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-24kn8" podStartSLOduration=2.862195444 podStartE2EDuration="22.595277095s" totalImagesPullingTime="19.733081651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:43.594996266 +0000 UTC m=+22.869965362" watchObservedRunningTime="2026-07-24 22:52:43.595277095 +0000 UTC m=+22.870246192" Jul 24 22:52:44.294451 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.294421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:44.294615 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:44.294587 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:44.395698 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.395655 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 22:52:44.505995 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.505958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" event={"ID":"aa302ee4376c6431a39ad601ac23daaa","Type":"ContainerStarted","Data":"d3c1de2c05c3714a1d39568b5cc46ad83ed9bbbdb4fdf08296044e4baf6f4b42"} Jul 24 22:52:44.509818 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.509757 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:52:44.510208 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.510183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"b14c67d39d8150fe84d4d1ec8a44e1ef7a3568284dbbe670c040cf526d5116f9"} Jul 24 22:52:44.512024 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.511993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cbl8q" event={"ID":"ff9e94c0-2b41-4a4c-9722-a5c65dbf688d","Type":"ContainerStarted","Data":"c765713408721a1e07e5cfaaf7704d10287d225ca2832e7a5336f25762c9021e"} Jul 24 22:52:44.514050 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.514025 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" event={"ID":"54cb923e-1fe5-4cef-a83a-50676bf4995b","Type":"ContainerStarted","Data":"1d936782222131406177eecf4f0119ca541e1ab66def5f2f35a71265c8c08e86"} Jul 24 22:52:44.523120 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.523071 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-241.ec2.internal" podStartSLOduration=22.523056238 podStartE2EDuration="22.523056238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:44.522896575 +0000 UTC m=+23.797865669" watchObservedRunningTime="2026-07-24 22:52:44.523056238 +0000 UTC m=+23.798025333" Jul 24 22:52:44.536351 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.536305 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cbl8q" podStartSLOduration=3.820426233 podStartE2EDuration="23.536288812s" totalImagesPullingTime="19.715862579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:44.536258002 +0000 UTC m=+23.811227097" watchObservedRunningTime="2026-07-24 22:52:44.536288812 +0000 UTC m=+23.811257907" Jul 24 22:52:44.975523 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.975490 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:44.976209 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:44.976150 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:45.164398 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.164302 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T22:52:44.39568645Z","UUID":"e59c8e3b-42ef-4c8a-93a1-d220c9a7c57e","Handler":null,"Name":"","Endpoint":""} Jul 24 22:52:45.165900 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.165875 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 22:52:45.166040 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.165922 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 22:52:45.297811 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.297735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:45.297811 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.297770 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:45.297993 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:45.297865 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:45.298039 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:45.297974 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:45.644269 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.644242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:45.644948 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:45.644920 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fz57w" Jul 24 22:52:46.294073 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:46.294027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:46.294236 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:46.294169 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:46.521468 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:46.521441 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:52:46.521906 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:46.521874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"7a863a125a451a9d967ae37d674c1d6175beef1a00908772231f97fea7dbe10c"} Jul 24 22:52:46.524226 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:46.524201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" event={"ID":"54cb923e-1fe5-4cef-a83a-50676bf4995b","Type":"ContainerStarted","Data":"8c02b054afc80d78d2acb37c819f06cad7cda6af18869953c8d3cdea53d41c28"} Jul 24 22:52:46.542917 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:46.542876 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9gsmw" podStartSLOduration=3.408586359 podStartE2EDuration="25.542861976s" totalImagesPullingTime="22.134275617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:46.542646834 +0000 UTC m=+25.817615954" watchObservedRunningTime="2026-07-24 22:52:46.542861976 +0000 UTC m=+25.817831070" Jul 24 22:52:47.294357 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:47.294112 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:47.294835 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:47.294152 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:47.294835 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:47.294463 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:47.294835 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:47.294540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:48.293996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.293974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:48.294103 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:48.294083 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:48.529624 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.529548 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0caf250-1643-481c-a696-e2e6dfe23b61" containerID="a1870b025471cff14afb01c603252da7f7c4652cb06272329b11b7e5276408ad" exitCode=0 Jul 24 22:52:48.530434 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.529638 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerDied","Data":"a1870b025471cff14afb01c603252da7f7c4652cb06272329b11b7e5276408ad"} Jul 24 22:52:48.533424 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.533226 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:52:48.533778 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.533755 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"a1ef43c3f167f9d44dbdbde54a141c86e09b4cfde4afaa639fa2b60bb2f734b0"} Jul 24 22:52:48.534039 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.534024 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:48.534186 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.534172 2562 scope.go:117] "RemoveContainer" containerID="95c29723973c9227c11a9d0aad66daec4da734b5e47bdc86aa1ff0ef68072a68" Jul 24 22:52:48.549961 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:48.549944 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:49.297415 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.297341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:49.297627 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:49.297439 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:49.297627 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.297502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:49.297764 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:49.297619 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:49.539688 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.539658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:52:49.540090 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.540036 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" event={"ID":"5dad8c34-9078-4e76-8f4c-5b1590840294","Type":"ContainerStarted","Data":"985fcde7bce730ef8b43dcba6c19867e990f5c82c6eb0b277e572c85cd7b76a5"} Jul 24 22:52:49.540353 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.540329 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:49.540514 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.540496 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:49.555181 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.555129 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:52:49.570379 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.570339 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" podStartSLOduration=8.197019478 podStartE2EDuration="28.570327254s" totalImagesPullingTime="20.373307776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:49.569889467 +0000 UTC m=+28.844858563" watchObservedRunningTime="2026-07-24 22:52:49.570327254 +0000 UTC m=+28.845296380" Jul 24 22:52:49.919882 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.919851 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c9rmb"] Jul 24 22:52:49.920035 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.920018 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:49.920162 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:49.920137 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:49.920505 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.920485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g75zv"] Jul 24 22:52:49.920587 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.920576 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:49.920720 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:49.920695 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:49.921932 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.921910 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jjhgr"] Jul 24 22:52:49.922020 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:49.922007 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:49.922120 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:49.922101 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:50.544346 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:50.544312 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0caf250-1643-481c-a696-e2e6dfe23b61" containerID="8161811b30aa687447a78b4de2c1229f3b0d80932011db3f105f68e36f7b9218" exitCode=0 Jul 24 22:52:50.544801 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:50.544405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerDied","Data":"8161811b30aa687447a78b4de2c1229f3b0d80932011db3f105f68e36f7b9218"} Jul 24 22:52:51.295217 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:51.295186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:51.295410 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:51.295302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:51.295410 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:51.295328 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:51.295410 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:51.295376 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:51.295410 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:51.295398 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:51.295636 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:51.295464 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:52.551023 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:52.550773 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0caf250-1643-481c-a696-e2e6dfe23b61" containerID="bf35585a9310d9dabcbbbc9cc6e97458c77419278b0071159f48989030d9c2bf" exitCode=0 Jul 24 22:52:52.551373 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:52.550865 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerDied","Data":"bf35585a9310d9dabcbbbc9cc6e97458c77419278b0071159f48989030d9c2bf"} Jul 24 22:52:53.294106 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:53.294067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:53.294318 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:53.294121 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:53.294318 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:53.294078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:53.294318 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.294204 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:53.294318 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.294267 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:53.294529 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.294332 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:53.959816 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:53.959780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:53.959849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:53.959878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.959941 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960001 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960016 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs podName:002ed68c-3e66-417f-9f2b-1b999ef83226 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:25.9599927 +0000 UTC m=+65.234961779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs") pod "network-metrics-daemon-c9rmb" (UID: "002ed68c-3e66-417f-9f2b-1b999ef83226") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960022 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960043 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960049 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret podName:d3e447b3-e0fe-4133-a696-e7d1c7f7358e nodeName:}" failed. No retries permitted until 2026-07-24 22:53:25.960032895 +0000 UTC m=+65.235001975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret") pod "global-pull-secret-syncer-g75zv" (UID: "d3e447b3-e0fe-4133-a696-e7d1c7f7358e") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960057 2562 projected.go:196] Error preparing data for projected volume kube-api-access-m69hd for pod openshift-network-diagnostics/network-check-target-jjhgr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:53.960244 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:53.960105 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd podName:6821a3e6-c3dc-4493-a05e-37306035e691 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:25.960090108 +0000 UTC m=+65.235059184 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m69hd" (UniqueName: "kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd") pod "network-check-target-jjhgr" (UID: "6821a3e6-c3dc-4493-a05e-37306035e691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:52:55.294222 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:55.294189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:55.294222 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:55.294217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:55.294863 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:55.294189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:55.294863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:55.294314 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jjhgr" podUID="6821a3e6-c3dc-4493-a05e-37306035e691" Jul 24 22:52:55.294863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:55.294394 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9rmb" podUID="002ed68c-3e66-417f-9f2b-1b999ef83226" Jul 24 22:52:55.294863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:55.294484 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g75zv" podUID="d3e447b3-e0fe-4133-a696-e7d1c7f7358e" Jul 24 22:52:56.553100 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.553026 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-241.ec2.internal" event="NodeReady" Jul 24 22:52:56.553574 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.553163 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 22:52:56.598481 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.598452 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sq92q"] Jul 24 22:52:56.601136 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.601099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.601298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.601275 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4hwnx"] Jul 24 22:52:56.602954 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.602934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.603960 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.603940 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 22:52:56.604073 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.604051 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 22:52:56.604287 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.604269 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4zlm\"" Jul 24 22:52:56.605689 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.605663 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z8ljk\"" Jul 24 22:52:56.606019 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.605947 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 22:52:56.606498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.606478 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 22:52:56.606979 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.606962 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 22:52:56.612893 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.612873 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4hwnx"] Jul 24 22:52:56.616839 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.616819 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sq92q"] Jul 24 22:52:56.779324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.779300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.779436 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.779339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c265bfe7-129d-4723-8348-3c9890868872-cert\") pod \"ingress-canary-4hwnx\" (UID: \"c265bfe7-129d-4723-8348-3c9890868872\") " pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.779436 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.779416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e800f15f-9f18-4c15-8baa-63cfaaee1138-config-volume\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.779556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.779444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2wbz\" (UniqueName: \"kubernetes.io/projected/e800f15f-9f18-4c15-8baa-63cfaaee1138-kube-api-access-z2wbz\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.779556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.779464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rft9t\" (UniqueName: \"kubernetes.io/projected/c265bfe7-129d-4723-8348-3c9890868872-kube-api-access-rft9t\") pod \"ingress-canary-4hwnx\" (UID: \"c265bfe7-129d-4723-8348-3c9890868872\") " pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.779556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.779499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e800f15f-9f18-4c15-8baa-63cfaaee1138-tmp-dir\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.880571 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e800f15f-9f18-4c15-8baa-63cfaaee1138-config-volume\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.880652 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2wbz\" (UniqueName: \"kubernetes.io/projected/e800f15f-9f18-4c15-8baa-63cfaaee1138-kube-api-access-z2wbz\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.880652 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rft9t\" (UniqueName: \"kubernetes.io/projected/c265bfe7-129d-4723-8348-3c9890868872-kube-api-access-rft9t\") pod \"ingress-canary-4hwnx\" (UID: \"c265bfe7-129d-4723-8348-3c9890868872\") " pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.880766 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e800f15f-9f18-4c15-8baa-63cfaaee1138-tmp-dir\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.880884 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.880949 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c265bfe7-129d-4723-8348-3c9890868872-cert\") pod \"ingress-canary-4hwnx\" (UID: \"c265bfe7-129d-4723-8348-3c9890868872\") " pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.881004 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:56.880984 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:52:56.881072 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:56.881061 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls podName:e800f15f-9f18-4c15-8baa-63cfaaee1138 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:57.381040223 +0000 UTC m=+36.656009298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls") pod "dns-default-sq92q" (UID: "e800f15f-9f18-4c15-8baa-63cfaaee1138") : secret "dns-default-metrics-tls" not found Jul 24 22:52:56.881121 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.880986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e800f15f-9f18-4c15-8baa-63cfaaee1138-tmp-dir\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.881159 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.881128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e800f15f-9f18-4c15-8baa-63cfaaee1138-config-volume\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.884845 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.884829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c265bfe7-129d-4723-8348-3c9890868872-cert\") pod \"ingress-canary-4hwnx\" (UID: \"c265bfe7-129d-4723-8348-3c9890868872\") " pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.889072 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.889054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2wbz\" (UniqueName: \"kubernetes.io/projected/e800f15f-9f18-4c15-8baa-63cfaaee1138-kube-api-access-z2wbz\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:56.889142 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.889106 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rft9t\" (UniqueName: \"kubernetes.io/projected/c265bfe7-129d-4723-8348-3c9890868872-kube-api-access-rft9t\") pod \"ingress-canary-4hwnx\" (UID: \"c265bfe7-129d-4723-8348-3c9890868872\") " pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:56.920257 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:56.920238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4hwnx" Jul 24 22:52:57.048494 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.048331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4hwnx"] Jul 24 22:52:57.052737 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:52:57.052713 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc265bfe7_129d_4723_8348_3c9890868872.slice/crio-963f479ee8af15045e5bf2bcd21279d1cf148b43870924372412566624a3adf5 WatchSource:0}: Error finding container 963f479ee8af15045e5bf2bcd21279d1cf148b43870924372412566624a3adf5: Status 404 returned error can't find the container with id 963f479ee8af15045e5bf2bcd21279d1cf148b43870924372412566624a3adf5 Jul 24 22:52:57.294094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.294001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:52:57.294094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.294021 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:52:57.294094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.294021 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:52:57.296968 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.296949 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:52:57.297063 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.297010 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:52:57.297063 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.297039 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:52:57.297821 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.297806 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2njwm\"" Jul 24 22:52:57.297879 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.297821 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:52:57.297879 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.297847 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9dffr\"" Jul 24 22:52:57.384574 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.384545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:57.384726 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:57.384692 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:52:57.384931 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:57.384760 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls podName:e800f15f-9f18-4c15-8baa-63cfaaee1138 nodeName:}" failed. No retries permitted until 2026-07-24 22:52:58.384739783 +0000 UTC m=+37.659708860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls") pod "dns-default-sq92q" (UID: "e800f15f-9f18-4c15-8baa-63cfaaee1138") : secret "dns-default-metrics-tls" not found Jul 24 22:52:57.567540 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.567450 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0caf250-1643-481c-a696-e2e6dfe23b61" containerID="351ec9ec90700baf05e649591a7c7d90b3ea68b94fb067a073eab8fe2dfab30f" exitCode=0 Jul 24 22:52:57.567540 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.567524 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerDied","Data":"351ec9ec90700baf05e649591a7c7d90b3ea68b94fb067a073eab8fe2dfab30f"} Jul 24 22:52:57.568896 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:57.568870 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4hwnx" event={"ID":"c265bfe7-129d-4723-8348-3c9890868872","Type":"ContainerStarted","Data":"963f479ee8af15045e5bf2bcd21279d1cf148b43870924372412566624a3adf5"} Jul 24 22:52:58.393086 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:58.393052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:52:58.393298 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:58.393225 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:52:58.393367 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:52:58.393305 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls podName:e800f15f-9f18-4c15-8baa-63cfaaee1138 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:00.393284132 +0000 UTC m=+39.668253208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls") pod "dns-default-sq92q" (UID: "e800f15f-9f18-4c15-8baa-63cfaaee1138") : secret "dns-default-metrics-tls" not found Jul 24 22:52:58.574383 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:58.574294 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0caf250-1643-481c-a696-e2e6dfe23b61" containerID="8b586b22a2be51efb1be9800bac7661c3d26a3c2cbb7a657b4d016f4544c046e" exitCode=0 Jul 24 22:52:58.574383 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:58.574343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerDied","Data":"8b586b22a2be51efb1be9800bac7661c3d26a3c2cbb7a657b4d016f4544c046e"} Jul 24 22:52:59.580098 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:59.579890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" event={"ID":"e0caf250-1643-481c-a696-e2e6dfe23b61","Type":"ContainerStarted","Data":"067d4f6d6ceb3db9c0807e932ee54fa4a04ff2a6859e579f924baf73d604f40c"} Jul 24 22:52:59.581376 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:59.581347 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4hwnx" event={"ID":"c265bfe7-129d-4723-8348-3c9890868872","Type":"ContainerStarted","Data":"9407bba33364ce344fd0d8e7fbf3a06e340f5969bb4d26d0f9060809a79b45f3"} Jul 24 22:52:59.601384 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:59.601344 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rdqk9" podStartSLOduration=7.685102157 podStartE2EDuration="38.601330014s" totalImagesPullingTime="30.916227857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:59.599925632 +0000 UTC m=+38.874894727" watchObservedRunningTime="2026-07-24 22:52:59.601330014 +0000 UTC m=+38.876299109" Jul 24 22:52:59.614428 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:52:59.614388 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4hwnx" podStartSLOduration=1.281382958 podStartE2EDuration="3.614374527s" totalImagesPullingTime="2.332991569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:59.614131646 +0000 UTC m=+38.889100743" watchObservedRunningTime="2026-07-24 22:52:59.614374527 +0000 UTC m=+38.889343619" Jul 24 22:53:00.411034 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:00.410994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:00.411225 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:00.411097 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:53:00.411225 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:00.411148 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls podName:e800f15f-9f18-4c15-8baa-63cfaaee1138 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:04.411135444 +0000 UTC m=+43.686104517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls") pod "dns-default-sq92q" (UID: "e800f15f-9f18-4c15-8baa-63cfaaee1138") : secret "dns-default-metrics-tls" not found Jul 24 22:53:01.489725 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.489693 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr"] Jul 24 22:53:01.511451 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.511423 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr"] Jul 24 22:53:01.511594 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.511536 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" Jul 24 22:53:01.514227 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.514205 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 22:53:01.514227 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.514213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:53:01.515335 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.515313 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dh6kk\"" Jul 24 22:53:01.584062 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.584027 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-785457d5c7-s7p6m"] Jul 24 22:53:01.607523 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.607495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-785457d5c7-s7p6m"] Jul 24 22:53:01.607696 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.607594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.610800 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.610777 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 22:53:01.610944 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.610804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 22:53:01.610944 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.610779 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 22:53:01.611111 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.611092 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 22:53:01.611225 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.611204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 22:53:01.611293 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.611205 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 22:53:01.611293 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.611236 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mxr6c\"" Jul 24 22:53:01.619896 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.619875 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwcz\" (UniqueName: \"kubernetes.io/projected/71ccb923-5f3c-4067-b7e2-e898e4841e81-kube-api-access-vnwcz\") pod \"volume-data-source-validator-549c6d54cf-wqjpr\" (UID: \"71ccb923-5f3c-4067-b7e2-e898e4841e81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" Jul 24 22:53:01.721125 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.721096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-stats-auth\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.721125 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.721126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.721319 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.721171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.721319 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.721194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgzfj\" (UniqueName: \"kubernetes.io/projected/5b272d32-e8bf-4812-b235-e425e33c7e06-kube-api-access-qgzfj\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.721319 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.721223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwcz\" (UniqueName: \"kubernetes.io/projected/71ccb923-5f3c-4067-b7e2-e898e4841e81-kube-api-access-vnwcz\") pod \"volume-data-source-validator-549c6d54cf-wqjpr\" (UID: \"71ccb923-5f3c-4067-b7e2-e898e4841e81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" Jul 24 22:53:01.721319 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.721242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-default-certificate\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.731517 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.731494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwcz\" (UniqueName: \"kubernetes.io/projected/71ccb923-5f3c-4067-b7e2-e898e4841e81-kube-api-access-vnwcz\") pod \"volume-data-source-validator-549c6d54cf-wqjpr\" (UID: \"71ccb923-5f3c-4067-b7e2-e898e4841e81\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" Jul 24 22:53:01.820745 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.820662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" Jul 24 22:53:01.821888 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.821864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-stats-auth\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.821982 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.821903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.822032 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.821997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.822032 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:01.822026 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:53:01.822147 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.822036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgzfj\" (UniqueName: \"kubernetes.io/projected/5b272d32-e8bf-4812-b235-e425e33c7e06-kube-api-access-qgzfj\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.822147 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:01.822094 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:02.322063945 +0000 UTC m=+41.597033020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : secret "router-metrics-certs-default" not found Jul 24 22:53:01.822147 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.822129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-default-certificate\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.822294 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:01.822183 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:02.322163958 +0000 UTC m=+41.597133031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : configmap references non-existent config key: service-ca.crt Jul 24 22:53:01.824323 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.824304 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-stats-auth\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.824406 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.824342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-default-certificate\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.830790 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.830771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgzfj\" (UniqueName: \"kubernetes.io/projected/5b272d32-e8bf-4812-b235-e425e33c7e06-kube-api-access-qgzfj\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:01.957928 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:01.957893 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr"] Jul 24 22:53:01.960742 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:01.960715 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ccb923_5f3c_4067_b7e2_e898e4841e81.slice/crio-8d12526d02c93f0cd1c57be51e9102d08fc5653fe37c15c9dfff3acfb5659b1f WatchSource:0}: Error finding container 8d12526d02c93f0cd1c57be51e9102d08fc5653fe37c15c9dfff3acfb5659b1f: Status 404 returned error can't find the container with id 8d12526d02c93f0cd1c57be51e9102d08fc5653fe37c15c9dfff3acfb5659b1f Jul 24 22:53:02.326721 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:02.326671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:02.326894 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:02.326738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:02.326894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:02.326768 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:53:02.326894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:02.326824 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:03.326809978 +0000 UTC m=+42.601779050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : secret "router-metrics-certs-default" not found Jul 24 22:53:02.326894 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:02.326844 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:03.326832962 +0000 UTC m=+42.601802035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : configmap references non-existent config key: service-ca.crt Jul 24 22:53:02.589764 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:02.589726 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" event={"ID":"71ccb923-5f3c-4067-b7e2-e898e4841e81","Type":"ContainerStarted","Data":"8d12526d02c93f0cd1c57be51e9102d08fc5653fe37c15c9dfff3acfb5659b1f"} Jul 24 22:53:03.335881 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:03.335841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:03.336044 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:03.335905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:03.336044 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:03.335977 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:53:03.336044 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:03.336037 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:05.336021419 +0000 UTC m=+44.610990496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : secret "router-metrics-certs-default" not found Jul 24 22:53:03.336173 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:03.336052 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:05.336046708 +0000 UTC m=+44.611015784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : configmap references non-existent config key: service-ca.crt Jul 24 22:53:04.444597 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:04.444556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:04.445059 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:04.444747 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:53:04.445059 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:04.444872 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls podName:e800f15f-9f18-4c15-8baa-63cfaaee1138 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:12.444849912 +0000 UTC m=+51.719818986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls") pod "dns-default-sq92q" (UID: "e800f15f-9f18-4c15-8baa-63cfaaee1138") : secret "dns-default-metrics-tls" not found Jul 24 22:53:04.596176 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:04.596141 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" event={"ID":"71ccb923-5f3c-4067-b7e2-e898e4841e81","Type":"ContainerStarted","Data":"c33702b43731266d2207de70e44dbcda7dc5740269562e7081b666c69bfdbed5"} Jul 24 22:53:04.612509 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:04.612463 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqjpr" podStartSLOduration=1.896590367 podStartE2EDuration="3.612451054s" totalImagesPullingTime="1.715860687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:04.610917073 +0000 UTC m=+43.885886170" watchObservedRunningTime="2026-07-24 22:53:04.612451054 +0000 UTC m=+43.887420181" Jul 24 22:53:05.349352 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:05.349309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:05.349519 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:05.349373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:05.349519 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:05.349455 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:53:05.349601 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:05.349524 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:09.349511589 +0000 UTC m=+48.624480674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : secret "router-metrics-certs-default" not found Jul 24 22:53:05.349601 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:05.349545 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:09.349530499 +0000 UTC m=+48.624499572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : configmap references non-existent config key: service-ca.crt Jul 24 22:53:06.235241 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.235210 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dncjt"] Jul 24 22:53:06.265616 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.265592 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dncjt"] Jul 24 22:53:06.265778 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.265738 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.268115 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.268093 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 22:53:06.268296 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.268160 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 22:53:06.268296 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.268165 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 22:53:06.269382 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.269361 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mzp5b\"" Jul 24 22:53:06.269496 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.269368 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 22:53:06.355298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.355265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22d61829-69c3-4f6e-9c91-26c8ac70b188-crio-socket\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.355298 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.355296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.355476 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.355319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.355476 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.355393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22d61829-69c3-4f6e-9c91-26c8ac70b188-data-volume\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.355476 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.355425 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jwn\" (UniqueName: \"kubernetes.io/projected/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-api-access-j6jwn\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456727 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.456695 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22d61829-69c3-4f6e-9c91-26c8ac70b188-crio-socket\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456727 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.456728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456915 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.456755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456915 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.456781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22d61829-69c3-4f6e-9c91-26c8ac70b188-data-volume\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456915 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.456796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jwn\" (UniqueName: \"kubernetes.io/projected/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-api-access-j6jwn\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456915 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.456835 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:53:06.456915 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.456887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/22d61829-69c3-4f6e-9c91-26c8ac70b188-crio-socket\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.456915 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.456901 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:53:06.457150 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.456912 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:06.9568907 +0000 UTC m=+46.231859781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : secret "insights-runtime-extractor-tls" not found Jul 24 22:53:06.457150 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.456990 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:06.956971517 +0000 UTC m=+46.231940596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : configmap "kube-rbac-proxy" not found Jul 24 22:53:06.457236 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.457159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/22d61829-69c3-4f6e-9c91-26c8ac70b188-data-volume\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.465669 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.465649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jwn\" (UniqueName: \"kubernetes.io/projected/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-api-access-j6jwn\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.960452 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.960416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.960452 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:06.960456 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:06.960652 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.960525 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:53:06.960652 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.960555 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:53:06.960652 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.960585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:07.960571875 +0000 UTC m=+47.235540948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : configmap "kube-rbac-proxy" not found Jul 24 22:53:06.960652 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:06.960604 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:07.960591753 +0000 UTC m=+47.235560830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : secret "insights-runtime-extractor-tls" not found Jul 24 22:53:07.967890 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:07.967857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:07.968323 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:07.967908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:07.968323 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:07.968005 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:53:07.968323 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:07.968070 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:09.968054286 +0000 UTC m=+49.243023360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : secret "insights-runtime-extractor-tls" not found Jul 24 22:53:07.968323 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:07.968008 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:53:07.968323 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:07.968165 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:09.968146395 +0000 UTC m=+49.243115469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : configmap "kube-rbac-proxy" not found Jul 24 22:53:08.234102 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.234026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6dnt_ce701a92-c7b7-43bf-8661-d2d5078f4836/dns-node-resolver/0.log" Jul 24 22:53:08.468254 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.468223 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw"] Jul 24 22:53:08.504613 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.504545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw"] Jul 24 22:53:08.504752 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.504630 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" Jul 24 22:53:08.507244 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.507221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ph94m\"" Jul 24 22:53:08.508425 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.508404 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 22:53:08.508550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.508457 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 22:53:08.573592 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.573566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74d4r\" (UniqueName: \"kubernetes.io/projected/042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a-kube-api-access-74d4r\") pod \"migrator-6865c9967-j7lzw\" (UID: \"042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" Jul 24 22:53:08.674490 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.674465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74d4r\" (UniqueName: \"kubernetes.io/projected/042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a-kube-api-access-74d4r\") pod \"migrator-6865c9967-j7lzw\" (UID: \"042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" Jul 24 22:53:08.681952 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.681933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74d4r\" (UniqueName: \"kubernetes.io/projected/042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a-kube-api-access-74d4r\") pod \"migrator-6865c9967-j7lzw\" (UID: \"042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" Jul 24 22:53:08.813534 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.813464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" Jul 24 22:53:08.938758 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:08.938731 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw"] Jul 24 22:53:08.941758 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:08.941721 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042186f1_d73b_4ba2_8c3e_b9ff3dbfd37a.slice/crio-4322f134931f2f4128abaae3247ac64c583cf756eb625ee7969d364db9d54451 WatchSource:0}: Error finding container 4322f134931f2f4128abaae3247ac64c583cf756eb625ee7969d364db9d54451: Status 404 returned error can't find the container with id 4322f134931f2f4128abaae3247ac64c583cf756eb625ee7969d364db9d54451 Jul 24 22:53:09.034199 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.034175 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-24kn8_a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6/node-ca/0.log" Jul 24 22:53:09.193234 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.193202 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bzphf"] Jul 24 22:53:09.226781 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.226755 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bzphf"] Jul 24 22:53:09.226907 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.226845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.229787 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.229751 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 22:53:09.229921 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.229841 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 22:53:09.229921 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.229913 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jzmt9\"" Jul 24 22:53:09.230796 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.230774 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 22:53:09.230894 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.230886 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 22:53:09.278734 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.278712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hm6\" (UniqueName: \"kubernetes.io/projected/205e49a8-bb0d-4131-89a6-590348911e4c-kube-api-access-k8hm6\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.278845 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.278798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/205e49a8-bb0d-4131-89a6-590348911e4c-signing-cabundle\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.278895 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.278883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/205e49a8-bb0d-4131-89a6-590348911e4c-signing-key\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.379181 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.379157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:09.379316 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.379203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/205e49a8-bb0d-4131-89a6-590348911e4c-signing-cabundle\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.379376 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:09.379318 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:17.379302453 +0000 UTC m=+56.654271526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : configmap references non-existent config key: service-ca.crt Jul 24 22:53:09.379448 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.379382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/205e49a8-bb0d-4131-89a6-590348911e4c-signing-key\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.379448 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.379419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hm6\" (UniqueName: \"kubernetes.io/projected/205e49a8-bb0d-4131-89a6-590348911e4c-kube-api-access-k8hm6\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.379550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.379451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:09.379550 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:09.379535 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:53:09.379646 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:09.379579 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:17.379566311 +0000 UTC m=+56.654535386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : secret "router-metrics-certs-default" not found Jul 24 22:53:09.379911 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.379891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/205e49a8-bb0d-4131-89a6-590348911e4c-signing-cabundle\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.381725 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.381706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/205e49a8-bb0d-4131-89a6-590348911e4c-signing-key\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.387580 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.387563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hm6\" (UniqueName: \"kubernetes.io/projected/205e49a8-bb0d-4131-89a6-590348911e4c-kube-api-access-k8hm6\") pod \"service-ca-658d7b7c78-bzphf\" (UID: \"205e49a8-bb0d-4131-89a6-590348911e4c\") " pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.535911 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.535828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" Jul 24 22:53:09.611590 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.611555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" event={"ID":"042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a","Type":"ContainerStarted","Data":"4322f134931f2f4128abaae3247ac64c583cf756eb625ee7969d364db9d54451"} Jul 24 22:53:09.676777 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.676734 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-bzphf"] Jul 24 22:53:09.680245 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:09.680207 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205e49a8_bb0d_4131_89a6_590348911e4c.slice/crio-d8e69e5818f93cec2f95e47d0f67d1b84c26a4f1cfb1e3b4aabf346372e90734 WatchSource:0}: Error finding container d8e69e5818f93cec2f95e47d0f67d1b84c26a4f1cfb1e3b4aabf346372e90734: Status 404 returned error can't find the container with id d8e69e5818f93cec2f95e47d0f67d1b84c26a4f1cfb1e3b4aabf346372e90734 Jul 24 22:53:09.834870 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.834837 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4hwnx_c265bfe7-129d-4723-8348-3c9890868872/serve-healthcheck-canary/0.log" Jul 24 22:53:09.984319 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.984290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:09.984542 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.984340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:09.984542 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:09.984446 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:53:09.984542 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:09.984508 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:13.984490941 +0000 UTC m=+53.259460026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : secret "insights-runtime-extractor-tls" not found Jul 24 22:53:09.984963 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:09.984942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/22d61829-69c3-4f6e-9c91-26c8ac70b188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:10.616196 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:10.616125 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" event={"ID":"042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a","Type":"ContainerStarted","Data":"d89953bb86f7c083eb998e5addffc24707fc67180697b2722be35afb9d795818"} Jul 24 22:53:10.617501 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:10.617457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" event={"ID":"205e49a8-bb0d-4131-89a6-590348911e4c","Type":"ContainerStarted","Data":"d8e69e5818f93cec2f95e47d0f67d1b84c26a4f1cfb1e3b4aabf346372e90734"} Jul 24 22:53:11.621709 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:11.621642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" event={"ID":"042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a","Type":"ContainerStarted","Data":"c6d99a0440622cbf47135a434e1528fd8c11b69fb796d8b8c672a6e9f38b3514"} Jul 24 22:53:11.638482 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:11.638427 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-j7lzw" podStartSLOduration=2.055625189 podStartE2EDuration="3.638407908s" totalImagesPullingTime="1.582782719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:11.638025795 +0000 UTC m=+50.912994891" watchObservedRunningTime="2026-07-24 22:53:11.638407908 +0000 UTC m=+50.913377004" Jul 24 22:53:12.504840 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:12.504815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:12.504991 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:12.504972 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:53:12.505039 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:12.505027 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls podName:e800f15f-9f18-4c15-8baa-63cfaaee1138 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:28.505013747 +0000 UTC m=+67.779982819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls") pod "dns-default-sq92q" (UID: "e800f15f-9f18-4c15-8baa-63cfaaee1138") : secret "dns-default-metrics-tls" not found Jul 24 22:53:12.626754 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:12.626651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" event={"ID":"205e49a8-bb0d-4131-89a6-590348911e4c","Type":"ContainerStarted","Data":"05ef8b0b2eaefd89ba874118127c217fbb2dd73f61c346c58ecd6cd9d5aaa5d4"} Jul 24 22:53:12.653348 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:12.653301 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-bzphf" podStartSLOduration=0.957190758 podStartE2EDuration="3.653285857s" totalImagesPullingTime="2.696095099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:12.652607682 +0000 UTC m=+51.927576777" watchObservedRunningTime="2026-07-24 22:53:12.653285857 +0000 UTC m=+51.928254970" Jul 24 22:53:14.017941 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:14.017897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:14.018410 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:14.018076 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:53:14.018410 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:14.018163 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls podName:22d61829-69c3-4f6e-9c91-26c8ac70b188 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:22.018142752 +0000 UTC m=+61.293111832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dncjt" (UID: "22d61829-69c3-4f6e-9c91-26c8ac70b188") : secret "insights-runtime-extractor-tls" not found Jul 24 22:53:17.447330 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:17.447279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:17.447863 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:17.447365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:17.447863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:17.447417 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:53:17.447863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:17.447490 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:33.447473326 +0000 UTC m=+72.722442400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : secret "router-metrics-certs-default" not found Jul 24 22:53:17.447863 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:53:17.447515 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle podName:5b272d32-e8bf-4812-b235-e425e33c7e06 nodeName:}" failed. No retries permitted until 2026-07-24 22:53:33.447500347 +0000 UTC m=+72.722469421 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle") pod "router-default-785457d5c7-s7p6m" (UID: "5b272d32-e8bf-4812-b235-e425e33c7e06") : configmap references non-existent config key: service-ca.crt Jul 24 22:53:21.564825 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:21.564797 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7wkj" Jul 24 22:53:22.080749 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:22.080705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:22.082954 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:22.082932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/22d61829-69c3-4f6e-9c91-26c8ac70b188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dncjt\" (UID: \"22d61829-69c3-4f6e-9c91-26c8ac70b188\") " pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:22.177063 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:22.177035 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mzp5b\"" Jul 24 22:53:22.185322 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:22.185300 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dncjt" Jul 24 22:53:22.303954 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:22.303925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dncjt"] Jul 24 22:53:22.306761 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:22.306732 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d61829_69c3_4f6e_9c91_26c8ac70b188.slice/crio-6b08be0f5d07336220b61863b20d80aeda1d373e2bf08c713c2c528e51de5ef2 WatchSource:0}: Error finding container 6b08be0f5d07336220b61863b20d80aeda1d373e2bf08c713c2c528e51de5ef2: Status 404 returned error can't find the container with id 6b08be0f5d07336220b61863b20d80aeda1d373e2bf08c713c2c528e51de5ef2 Jul 24 22:53:22.657974 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:22.657939 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dncjt" event={"ID":"22d61829-69c3-4f6e-9c91-26c8ac70b188","Type":"ContainerStarted","Data":"6b08be0f5d07336220b61863b20d80aeda1d373e2bf08c713c2c528e51de5ef2"} Jul 24 22:53:26.013584 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.013540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:53:26.013584 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.013585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:53:26.014132 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.013658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:53:26.016526 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.016498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:53:26.017581 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.017559 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:53:26.017664 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.017563 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:53:26.027139 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.027110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3e447b3-e0fe-4133-a696-e7d1c7f7358e-original-pull-secret\") pod \"global-pull-secret-syncer-g75zv\" (UID: \"d3e447b3-e0fe-4133-a696-e7d1c7f7358e\") " pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:53:26.027251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.027204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:53:26.027251 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.027222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002ed68c-3e66-417f-9f2b-1b999ef83226-metrics-certs\") pod \"network-metrics-daemon-c9rmb\" (UID: \"002ed68c-3e66-417f-9f2b-1b999ef83226\") " pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:53:26.037743 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.037725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69hd\" (UniqueName: \"kubernetes.io/projected/6821a3e6-c3dc-4493-a05e-37306035e691-kube-api-access-m69hd\") pod \"network-check-target-jjhgr\" (UID: \"6821a3e6-c3dc-4493-a05e-37306035e691\") " pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:53:26.106603 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.106564 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9dffr\"" Jul 24 22:53:26.109640 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.109609 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g75zv" Jul 24 22:53:26.113954 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.113913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:53:26.116094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.116072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2njwm\"" Jul 24 22:53:26.124320 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.124297 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9rmb" Jul 24 22:53:26.838383 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:26.838354 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c9rmb"] Jul 24 22:53:27.054754 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:27.054539 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002ed68c_3e66_417f_9f2b_1b999ef83226.slice/crio-14b319d05739d321111ee471e927fe6a0f6ac0a7d83da9e69129d88611fb752d WatchSource:0}: Error finding container 14b319d05739d321111ee471e927fe6a0f6ac0a7d83da9e69129d88611fb752d: Status 404 returned error can't find the container with id 14b319d05739d321111ee471e927fe6a0f6ac0a7d83da9e69129d88611fb752d Jul 24 22:53:27.175406 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.175381 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g75zv"] Jul 24 22:53:27.178015 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:27.177986 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e447b3_e0fe_4133_a696_e7d1c7f7358e.slice/crio-24627217d726026c4f257b34da9d21beec6074ec04ed7887a28962abeca89f2c WatchSource:0}: Error finding container 24627217d726026c4f257b34da9d21beec6074ec04ed7887a28962abeca89f2c: Status 404 returned error can't find the container with id 24627217d726026c4f257b34da9d21beec6074ec04ed7887a28962abeca89f2c Jul 24 22:53:27.188604 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.188579 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jjhgr"] Jul 24 22:53:27.191248 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:27.191225 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6821a3e6_c3dc_4493_a05e_37306035e691.slice/crio-7c27dea2e67e77f52df62097f7759d5312b36516d737dbd1ff854fd5c589a6a3 WatchSource:0}: Error finding container 7c27dea2e67e77f52df62097f7759d5312b36516d737dbd1ff854fd5c589a6a3: Status 404 returned error can't find the container with id 7c27dea2e67e77f52df62097f7759d5312b36516d737dbd1ff854fd5c589a6a3 Jul 24 22:53:27.336545 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.336520 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9"] Jul 24 22:53:27.339556 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.339541 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.342785 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.342762 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 22:53:27.342982 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.342965 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 22:53:27.343633 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.343615 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 22:53:27.344204 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.344177 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 22:53:27.344572 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.344556 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 22:53:27.344659 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.344580 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 22:53:27.344730 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.344655 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 22:53:27.356526 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.356504 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9"] Jul 24 22:53:27.424020 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.423997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.424147 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.424053 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.424147 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.424130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-ca\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.424244 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.424159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac780ec5-c7bb-4f9d-afd9-d38848731177-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.424244 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.424184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-hub\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.424244 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.424210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zwn\" (UniqueName: \"kubernetes.io/projected/ac780ec5-c7bb-4f9d-afd9-d38848731177-kube-api-access-v8zwn\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.525543 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.525513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.525745 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.525573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-ca\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.525745 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.525595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac780ec5-c7bb-4f9d-afd9-d38848731177-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.525745 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.525618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-hub\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.525745 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.525640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zwn\" (UniqueName: \"kubernetes.io/projected/ac780ec5-c7bb-4f9d-afd9-d38848731177-kube-api-access-v8zwn\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.525745 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.525689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.526595 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.526566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac780ec5-c7bb-4f9d-afd9-d38848731177-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.528512 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.528491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-ca\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.528621 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.528605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.528689 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.528654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-hub\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.528738 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.528656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ac780ec5-c7bb-4f9d-afd9-d38848731177-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.534114 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.534095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zwn\" (UniqueName: \"kubernetes.io/projected/ac780ec5-c7bb-4f9d-afd9-d38848731177-kube-api-access-v8zwn\") pod \"cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9\" (UID: \"ac780ec5-c7bb-4f9d-afd9-d38848731177\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.661366 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.661277 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" Jul 24 22:53:27.680882 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.680853 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c9rmb" event={"ID":"002ed68c-3e66-417f-9f2b-1b999ef83226","Type":"ContainerStarted","Data":"14b319d05739d321111ee471e927fe6a0f6ac0a7d83da9e69129d88611fb752d"} Jul 24 22:53:27.685309 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.685231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dncjt" event={"ID":"22d61829-69c3-4f6e-9c91-26c8ac70b188","Type":"ContainerStarted","Data":"0facbd8bc9858ef029ea66cc6b36a7651e930043239d95bb8a2b097d0c9ff679"} Jul 24 22:53:27.694191 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.692246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jjhgr" event={"ID":"6821a3e6-c3dc-4493-a05e-37306035e691","Type":"ContainerStarted","Data":"7c27dea2e67e77f52df62097f7759d5312b36516d737dbd1ff854fd5c589a6a3"} Jul 24 22:53:27.696699 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.695369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g75zv" event={"ID":"d3e447b3-e0fe-4133-a696-e7d1c7f7358e","Type":"ContainerStarted","Data":"24627217d726026c4f257b34da9d21beec6074ec04ed7887a28962abeca89f2c"} Jul 24 22:53:27.831457 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:27.831343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9"] Jul 24 22:53:27.836587 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:27.836556 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac780ec5_c7bb_4f9d_afd9_d38848731177.slice/crio-97e3fd5fa62c421ba9977a8d9978df73b7f9f1d06441ca5c49ebd2662c758c0e WatchSource:0}: Error finding container 97e3fd5fa62c421ba9977a8d9978df73b7f9f1d06441ca5c49ebd2662c758c0e: Status 404 returned error can't find the container with id 97e3fd5fa62c421ba9977a8d9978df73b7f9f1d06441ca5c49ebd2662c758c0e Jul 24 22:53:28.534826 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:28.534785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:28.543099 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:28.543070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e800f15f-9f18-4c15-8baa-63cfaaee1138-metrics-tls\") pod \"dns-default-sq92q\" (UID: \"e800f15f-9f18-4c15-8baa-63cfaaee1138\") " pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:28.702539 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:28.702477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" event={"ID":"ac780ec5-c7bb-4f9d-afd9-d38848731177","Type":"ContainerStarted","Data":"97e3fd5fa62c421ba9977a8d9978df73b7f9f1d06441ca5c49ebd2662c758c0e"} Jul 24 22:53:28.717331 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:28.717131 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4zlm\"" Jul 24 22:53:28.725252 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:28.725038 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:28.953436 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:28.953347 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sq92q"] Jul 24 22:53:28.956768 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:28.956734 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode800f15f_9f18_4c15_8baa_63cfaaee1138.slice/crio-c6161fd2d9fce7169eae7da3f9a8d13e91c0ec82cbfca38df6797d915ade4917 WatchSource:0}: Error finding container c6161fd2d9fce7169eae7da3f9a8d13e91c0ec82cbfca38df6797d915ade4917: Status 404 returned error can't find the container with id c6161fd2d9fce7169eae7da3f9a8d13e91c0ec82cbfca38df6797d915ade4917 Jul 24 22:53:29.712282 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:29.712156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sq92q" event={"ID":"e800f15f-9f18-4c15-8baa-63cfaaee1138","Type":"ContainerStarted","Data":"c6161fd2d9fce7169eae7da3f9a8d13e91c0ec82cbfca38df6797d915ade4917"} Jul 24 22:53:29.719345 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:29.719209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c9rmb" event={"ID":"002ed68c-3e66-417f-9f2b-1b999ef83226","Type":"ContainerStarted","Data":"243368100e60254ddb10f1df1afa2cd0b24babae6e5ef45921875ecc5c2dabbc"} Jul 24 22:53:29.719345 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:29.719313 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c9rmb" event={"ID":"002ed68c-3e66-417f-9f2b-1b999ef83226","Type":"ContainerStarted","Data":"7e1f99d8cf88e593dc34d22244788a20625e47d4e6af1a30ab13c6fcc3b8a131"} Jul 24 22:53:33.478465 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:33.478428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:33.478987 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:33.478478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:33.479191 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:33.479169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b272d32-e8bf-4812-b235-e425e33c7e06-service-ca-bundle\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:33.481103 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:33.481076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b272d32-e8bf-4812-b235-e425e33c7e06-metrics-certs\") pod \"router-default-785457d5c7-s7p6m\" (UID: \"5b272d32-e8bf-4812-b235-e425e33c7e06\") " pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:33.719560 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:33.719527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mxr6c\"" Jul 24 22:53:33.726814 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:33.726793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:36.705540 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.704509 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c9rmb" podStartSLOduration=74.065378159 podStartE2EDuration="1m15.704481082s" totalImagesPullingTime="1.639102923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:29.736271714 +0000 UTC m=+69.011240811" watchObservedRunningTime="2026-07-24 22:53:36.704481082 +0000 UTC m=+75.979450179" Jul 24 22:53:36.705540 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.705480 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-785457d5c7-s7p6m"] Jul 24 22:53:36.716153 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:36.716120 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b272d32_e8bf_4812_b235_e425e33c7e06.slice/crio-2c84d2ddb422d794d83a624096246a9369643b8f24c667ee1a401be46547d3fa WatchSource:0}: Error finding container 2c84d2ddb422d794d83a624096246a9369643b8f24c667ee1a401be46547d3fa: Status 404 returned error can't find the container with id 2c84d2ddb422d794d83a624096246a9369643b8f24c667ee1a401be46547d3fa Jul 24 22:53:36.748189 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.748138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sq92q" event={"ID":"e800f15f-9f18-4c15-8baa-63cfaaee1138","Type":"ContainerStarted","Data":"4c3a8d86481bf9090565f28da2f108214e99590c5d7c1aae6325ee692f3d198c"} Jul 24 22:53:36.750078 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.750050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dncjt" event={"ID":"22d61829-69c3-4f6e-9c91-26c8ac70b188","Type":"ContainerStarted","Data":"bf3c95d7d0004193f1541a908bb53b77084ae3c193398a9d338e75d240f8d7e8"} Jul 24 22:53:36.751835 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.751814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" event={"ID":"ac780ec5-c7bb-4f9d-afd9-d38848731177","Type":"ContainerStarted","Data":"442c8952a44226c7e4f657cdbdffc33e502c651b53b2f930043a413349de1cd7"} Jul 24 22:53:36.754217 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.753536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jjhgr" event={"ID":"6821a3e6-c3dc-4493-a05e-37306035e691","Type":"ContainerStarted","Data":"1d0ffbfe89f8034069425d12910c55a6596580a12b7765c99a099d9e025293e7"} Jul 24 22:53:36.754217 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.754175 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:53:36.755946 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.755924 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g75zv" event={"ID":"d3e447b3-e0fe-4133-a696-e7d1c7f7358e","Type":"ContainerStarted","Data":"2bb37cf33d1eb10daddf2231752f1f5070594bbe1f5e162f33ddb95784364901"} Jul 24 22:53:36.756996 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.756976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-785457d5c7-s7p6m" event={"ID":"5b272d32-e8bf-4812-b235-e425e33c7e06","Type":"ContainerStarted","Data":"2c84d2ddb422d794d83a624096246a9369643b8f24c667ee1a401be46547d3fa"} Jul 24 22:53:36.770034 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.769830 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jjhgr" podStartSLOduration=66.410670704 podStartE2EDuration="1m15.769811323s" totalImagesPullingTime="9.359140619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:36.769087975 +0000 UTC m=+76.044057071" watchObservedRunningTime="2026-07-24 22:53:36.769811323 +0000 UTC m=+76.044780419" Jul 24 22:53:36.786767 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:36.786260 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g75zv" podStartSLOduration=66.413896521 podStartE2EDuration="1m15.786241303s" totalImagesPullingTime="9.372344782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:36.785797326 +0000 UTC m=+76.060766423" watchObservedRunningTime="2026-07-24 22:53:36.786241303 +0000 UTC m=+76.061210403" Jul 24 22:53:37.762950 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.762903 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sq92q" event={"ID":"e800f15f-9f18-4c15-8baa-63cfaaee1138","Type":"ContainerStarted","Data":"14b46ce38daa0f8c64731381e1464872f1bc257a8eed9e460d4372f3ad0d78be"} Jul 24 22:53:37.763428 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.763003 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:37.765222 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.765195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" event={"ID":"ac780ec5-c7bb-4f9d-afd9-d38848731177","Type":"ContainerStarted","Data":"76d079914fd7384bbaf2f4d9ca3713d81492a5be624d5100cc6044941b521bcc"} Jul 24 22:53:37.765350 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.765233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" event={"ID":"ac780ec5-c7bb-4f9d-afd9-d38848731177","Type":"ContainerStarted","Data":"9dc155b78d8b1b2be9460bdc1ddab23d1d7cde207964827fb2239e957929eb0e"} Jul 24 22:53:37.766638 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.766614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-785457d5c7-s7p6m" event={"ID":"5b272d32-e8bf-4812-b235-e425e33c7e06","Type":"ContainerStarted","Data":"5ff47ca60eac25a7b8055d449b9927b1b5ecf245bd3780ff0b8ca17d285025bb"} Jul 24 22:53:37.781517 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.781472 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sq92q" podStartSLOduration=34.187924701 podStartE2EDuration="41.781459485s" totalImagesPullingTime="7.593534784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:37.778813373 +0000 UTC m=+77.053782469" watchObservedRunningTime="2026-07-24 22:53:37.781459485 +0000 UTC m=+77.056428579" Jul 24 22:53:37.796933 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.796756 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6db4ffdbf5-hfcj9" podStartSLOduration=2.061806932 podStartE2EDuration="10.796743849s" totalImagesPullingTime="8.734936917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:37.796247802 +0000 UTC m=+77.071216906" watchObservedRunningTime="2026-07-24 22:53:37.796743849 +0000 UTC m=+77.071712942" Jul 24 22:53:37.818687 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:37.818636 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-785457d5c7-s7p6m" podStartSLOduration=36.818620961 podStartE2EDuration="36.818620961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:37.81735943 +0000 UTC m=+77.092328524" watchObservedRunningTime="2026-07-24 22:53:37.818620961 +0000 UTC m=+77.093590055" Jul 24 22:53:38.727446 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:38.727413 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:38.730139 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:38.730109 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:38.771527 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:38.771498 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:38.772343 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:38.772320 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-785457d5c7-s7p6m" Jul 24 22:53:40.780050 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:40.780017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dncjt" event={"ID":"22d61829-69c3-4f6e-9c91-26c8ac70b188","Type":"ContainerStarted","Data":"00a55df466199b2b2eed6d4751615ae39185ade61f940238db8d41c6f0acaf27"} Jul 24 22:53:40.798818 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:40.798775 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dncjt" podStartSLOduration=17.263485782 podStartE2EDuration="34.798762444s" totalImagesPullingTime="17.535276662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:53:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:40.797728482 +0000 UTC m=+80.072697576" watchObservedRunningTime="2026-07-24 22:53:40.798762444 +0000 UTC m=+80.073731540" Jul 24 22:53:47.492534 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.492396 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qjjq4"] Jul 24 22:53:47.496970 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.496944 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.499796 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.499767 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 22:53:47.499910 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.499807 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 22:53:47.500229 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.500205 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 22:53:47.500307 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.500238 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 22:53:47.501326 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.501301 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 22:53:47.501603 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.501584 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbwd8\"" Jul 24 22:53:47.501706 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.501633 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 22:53:47.585470 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-wtmp\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585636 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585636 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-root\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585636 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwmz\" (UniqueName: \"kubernetes.io/projected/e100cbed-1739-4a89-85b6-4b7ff1daef3f-kube-api-access-cvwmz\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585636 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-textfile\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585636 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-tls\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585908 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585908 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e100cbed-1739-4a89-85b6-4b7ff1daef3f-metrics-client-ca\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.585908 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.585707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-sys\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.686685 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.686847 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e100cbed-1739-4a89-85b6-4b7ff1daef3f-metrics-client-ca\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.686847 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-sys\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.686847 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-wtmp\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.686847 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.686847 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-root\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-sys\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-root\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwmz\" (UniqueName: \"kubernetes.io/projected/e100cbed-1739-4a89-85b6-4b7ff1daef3f-kube-api-access-cvwmz\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-textfile\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.686974 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-wtmp\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.687008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-tls\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687310 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.687265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-textfile\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687444 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.687420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-accelerators-collector-config\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.687540 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.687525 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e100cbed-1739-4a89-85b6-4b7ff1daef3f-metrics-client-ca\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.689056 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.689036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.689272 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.689255 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e100cbed-1739-4a89-85b6-4b7ff1daef3f-node-exporter-tls\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.695789 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.695769 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwmz\" (UniqueName: \"kubernetes.io/projected/e100cbed-1739-4a89-85b6-4b7ff1daef3f-kube-api-access-cvwmz\") pod \"node-exporter-qjjq4\" (UID: \"e100cbed-1739-4a89-85b6-4b7ff1daef3f\") " pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.773254 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.773183 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sq92q" Jul 24 22:53:47.807746 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:47.807718 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qjjq4" Jul 24 22:53:47.819316 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:47.819286 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode100cbed_1739_4a89_85b6_4b7ff1daef3f.slice/crio-fcdadf1d56bcd4a54f4425f872cf6441c74248cea348a981e399faef0d5c8f50 WatchSource:0}: Error finding container fcdadf1d56bcd4a54f4425f872cf6441c74248cea348a981e399faef0d5c8f50: Status 404 returned error can't find the container with id fcdadf1d56bcd4a54f4425f872cf6441c74248cea348a981e399faef0d5c8f50 Jul 24 22:53:48.807117 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:48.807082 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qjjq4" event={"ID":"e100cbed-1739-4a89-85b6-4b7ff1daef3f","Type":"ContainerStarted","Data":"fcdadf1d56bcd4a54f4425f872cf6441c74248cea348a981e399faef0d5c8f50"} Jul 24 22:53:49.812438 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:49.812404 2562 generic.go:356] "Generic (PLEG): container finished" podID="e100cbed-1739-4a89-85b6-4b7ff1daef3f" containerID="54cf1b15c0efe7db5f64de68928c65a63ef57e64db0f3e262c236978c500c952" exitCode=0 Jul 24 22:53:49.812813 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:49.812473 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qjjq4" event={"ID":"e100cbed-1739-4a89-85b6-4b7ff1daef3f","Type":"ContainerDied","Data":"54cf1b15c0efe7db5f64de68928c65a63ef57e64db0f3e262c236978c500c952"} Jul 24 22:53:50.818352 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:50.818321 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qjjq4" event={"ID":"e100cbed-1739-4a89-85b6-4b7ff1daef3f","Type":"ContainerStarted","Data":"717fcdbd7e967521fcfb9af7d1bdf44da47070d5e86fbbf6c554e861661e911d"} Jul 24 22:53:50.818718 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:50.818368 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qjjq4" event={"ID":"e100cbed-1739-4a89-85b6-4b7ff1daef3f","Type":"ContainerStarted","Data":"91c6178d84f0aeac59166ea4ec8b5de9f7043f95ae0b1fc77fb5ca97639138fd"} Jul 24 22:53:50.842140 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:50.842082 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qjjq4" podStartSLOduration=2.779913218 podStartE2EDuration="3.842067277s" totalImagesPullingTime="1.000494449s" totalInitContainerRuntime="61.65961ms" isStatefulPod=true podCreationTimestamp="2026-07-24 22:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:50.840050822 +0000 UTC m=+90.115019943" watchObservedRunningTime="2026-07-24 22:53:50.842067277 +0000 UTC m=+90.117036371" Jul 24 22:53:53.919353 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.919314 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cf884bcdf-xttpg"] Jul 24 22:53:53.922545 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.922523 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:53.924971 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.924943 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 22:53:53.925077 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.925057 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 22:53:53.925234 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.925216 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fjmtg\"" Jul 24 22:53:53.925293 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.925237 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 22:53:53.926236 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.926218 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 22:53:53.926370 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.926304 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 22:53:53.926370 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.926327 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 22:53:53.926370 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.926333 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 22:53:53.932216 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:53.932196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf884bcdf-xttpg"] Jul 24 22:53:54.032903 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.032876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-oauth-config\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.033025 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.032942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-service-ca\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.033025 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.032967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-serving-cert\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.033025 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.032982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-oauth-serving-cert\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.033025 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.033001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27nj\" (UniqueName: \"kubernetes.io/projected/41c7fb8b-a604-4a7f-8d44-222450b6fab6-kube-api-access-p27nj\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.033170 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.033027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-config\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.133972 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.133943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-service-ca\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.133972 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.133975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-serving-cert\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.134152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.133993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-oauth-serving-cert\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.134152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.134014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27nj\" (UniqueName: \"kubernetes.io/projected/41c7fb8b-a604-4a7f-8d44-222450b6fab6-kube-api-access-p27nj\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.134152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.134032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-config\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.134152 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.134062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-oauth-config\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.134884 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.134860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-config\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.135001 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.134859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-service-ca\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.135001 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.134866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-oauth-serving-cert\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.136981 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.136960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-serving-cert\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.137151 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.137135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-oauth-config\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.145516 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.145496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27nj\" (UniqueName: \"kubernetes.io/projected/41c7fb8b-a604-4a7f-8d44-222450b6fab6-kube-api-access-p27nj\") pod \"console-5cf884bcdf-xttpg\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.232028 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.231939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:53:54.358434 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.358404 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf884bcdf-xttpg"] Jul 24 22:53:54.361523 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:53:54.361499 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c7fb8b_a604_4a7f_8d44_222450b6fab6.slice/crio-0f13f654e824f5e3e958615d3bb40aa8e30dc1a174b84d007f5691a3c723419a WatchSource:0}: Error finding container 0f13f654e824f5e3e958615d3bb40aa8e30dc1a174b84d007f5691a3c723419a: Status 404 returned error can't find the container with id 0f13f654e824f5e3e958615d3bb40aa8e30dc1a174b84d007f5691a3c723419a Jul 24 22:53:54.832471 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:54.832439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf884bcdf-xttpg" event={"ID":"41c7fb8b-a604-4a7f-8d44-222450b6fab6","Type":"ContainerStarted","Data":"0f13f654e824f5e3e958615d3bb40aa8e30dc1a174b84d007f5691a3c723419a"} Jul 24 22:53:57.845789 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:57.845755 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf884bcdf-xttpg" event={"ID":"41c7fb8b-a604-4a7f-8d44-222450b6fab6","Type":"ContainerStarted","Data":"9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c"} Jul 24 22:53:57.863089 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:53:57.863042 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cf884bcdf-xttpg" podStartSLOduration=2.012618194 podStartE2EDuration="4.863028163s" totalImagesPullingTime="2.850409969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:57.862169757 +0000 UTC m=+97.137138851" watchObservedRunningTime="2026-07-24 22:53:57.863028163 +0000 UTC m=+97.137997258" Jul 24 22:54:04.232124 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:04.232075 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:54:04.232587 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:04.232147 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:54:04.236911 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:04.236888 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:54:04.873166 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:04.873138 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:54:08.773900 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:08.773869 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jjhgr" Jul 24 22:54:15.349162 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:15.349129 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf884bcdf-xttpg"] Jul 24 22:54:40.368003 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.367954 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cf884bcdf-xttpg" podUID="41c7fb8b-a604-4a7f-8d44-222450b6fab6" containerName="console" containerID="cri-o://9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c" gracePeriod=15 Jul 24 22:54:40.607816 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.607790 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:54:40.663865 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.663786 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p27nj\" (UniqueName: \"kubernetes.io/projected/41c7fb8b-a604-4a7f-8d44-222450b6fab6-kube-api-access-p27nj\") pod \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " Jul 24 22:54:40.663865 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.663843 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-service-ca\") pod \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " Jul 24 22:54:40.664066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.663877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-oauth-config\") pod \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " Jul 24 22:54:40.664066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.663893 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-serving-cert\") pod \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " Jul 24 22:54:40.664066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.663910 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-oauth-serving-cert\") pod \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " Jul 24 22:54:40.664066 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.663930 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-config\") pod \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\" (UID: \"41c7fb8b-a604-4a7f-8d44-222450b6fab6\") " Jul 24 22:54:40.664324 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.664295 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-service-ca" (OuterVolumeSpecName: "service-ca") pod "41c7fb8b-a604-4a7f-8d44-222450b6fab6" (UID: "41c7fb8b-a604-4a7f-8d44-222450b6fab6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:54:40.664397 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.664314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41c7fb8b-a604-4a7f-8d44-222450b6fab6" (UID: "41c7fb8b-a604-4a7f-8d44-222450b6fab6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:54:40.664397 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.664343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-config" (OuterVolumeSpecName: "console-config") pod "41c7fb8b-a604-4a7f-8d44-222450b6fab6" (UID: "41c7fb8b-a604-4a7f-8d44-222450b6fab6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:54:40.666086 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.666056 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41c7fb8b-a604-4a7f-8d44-222450b6fab6" (UID: "41c7fb8b-a604-4a7f-8d44-222450b6fab6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:54:40.666186 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.666099 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c7fb8b-a604-4a7f-8d44-222450b6fab6-kube-api-access-p27nj" (OuterVolumeSpecName: "kube-api-access-p27nj") pod "41c7fb8b-a604-4a7f-8d44-222450b6fab6" (UID: "41c7fb8b-a604-4a7f-8d44-222450b6fab6"). InnerVolumeSpecName "kube-api-access-p27nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:54:40.666186 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.666120 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41c7fb8b-a604-4a7f-8d44-222450b6fab6" (UID: "41c7fb8b-a604-4a7f-8d44-222450b6fab6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:54:40.765267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.765219 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p27nj\" (UniqueName: \"kubernetes.io/projected/41c7fb8b-a604-4a7f-8d44-222450b6fab6-kube-api-access-p27nj\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:54:40.765267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.765261 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:54:40.765267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.765275 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-oauth-config\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:54:40.765478 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.765290 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-serving-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:54:40.765478 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.765302 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-oauth-serving-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:54:40.765478 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.765313 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41c7fb8b-a604-4a7f-8d44-222450b6fab6-console-config\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:54:40.997400 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.997316 2562 generic.go:356] "Generic (PLEG): container finished" podID="41c7fb8b-a604-4a7f-8d44-222450b6fab6" containerID="9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c" exitCode=0 Jul 24 22:54:40.997400 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.997366 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf884bcdf-xttpg" Jul 24 22:54:40.997400 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.997373 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf884bcdf-xttpg" event={"ID":"41c7fb8b-a604-4a7f-8d44-222450b6fab6","Type":"ContainerDied","Data":"9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c"} Jul 24 22:54:40.997618 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.997421 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf884bcdf-xttpg" event={"ID":"41c7fb8b-a604-4a7f-8d44-222450b6fab6","Type":"ContainerDied","Data":"0f13f654e824f5e3e958615d3bb40aa8e30dc1a174b84d007f5691a3c723419a"} Jul 24 22:54:40.997618 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:40.997449 2562 scope.go:117] "RemoveContainer" containerID="9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c" Jul 24 22:54:41.006146 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:41.006111 2562 scope.go:117] "RemoveContainer" containerID="9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c" Jul 24 22:54:41.006449 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:54:41.006424 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c\": container with ID starting with 9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c not found: ID does not exist" containerID="9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c" Jul 24 22:54:41.006525 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:41.006463 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c"} err="failed to get container status \"9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c\": rpc error: code = NotFound desc = could not find container \"9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c\": container with ID starting with 9b866824aeea1e53a5c33c890d6b79a0a8f31ae20935d4b7426047f0ef8c7f8c not found: ID does not exist" Jul 24 22:54:41.018094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:41.018072 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf884bcdf-xttpg"] Jul 24 22:54:41.022040 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:41.022020 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cf884bcdf-xttpg"] Jul 24 22:54:41.298943 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:54:41.298862 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c7fb8b-a604-4a7f-8d44-222450b6fab6" path="/var/lib/kubelet/pods/41c7fb8b-a604-4a7f-8d44-222450b6fab6/volumes" Jul 24 22:56:43.152039 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.151965 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s"] Jul 24 22:56:43.152475 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.152211 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c7fb8b-a604-4a7f-8d44-222450b6fab6" containerName="console" Jul 24 22:56:43.152475 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.152222 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c7fb8b-a604-4a7f-8d44-222450b6fab6" containerName="console" Jul 24 22:56:43.152475 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.152273 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c7fb8b-a604-4a7f-8d44-222450b6fab6" containerName="console" Jul 24 22:56:43.154809 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.154792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.157652 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.157630 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:56:43.158635 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.158615 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jj5dv\"" Jul 24 22:56:43.158635 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.158630 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 22:56:43.162386 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.162357 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s"] Jul 24 22:56:43.186196 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.186171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27aba347-8a24-4cd1-a1dc-1d6b909944a1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nbk7s\" (UID: \"27aba347-8a24-4cd1-a1dc-1d6b909944a1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.186303 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.186237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwcp\" (UniqueName: \"kubernetes.io/projected/27aba347-8a24-4cd1-a1dc-1d6b909944a1-kube-api-access-njwcp\") pod \"openshift-lws-operator-fd8ccff4c-nbk7s\" (UID: \"27aba347-8a24-4cd1-a1dc-1d6b909944a1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.286633 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.286601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwcp\" (UniqueName: \"kubernetes.io/projected/27aba347-8a24-4cd1-a1dc-1d6b909944a1-kube-api-access-njwcp\") pod \"openshift-lws-operator-fd8ccff4c-nbk7s\" (UID: \"27aba347-8a24-4cd1-a1dc-1d6b909944a1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.286800 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.286646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27aba347-8a24-4cd1-a1dc-1d6b909944a1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nbk7s\" (UID: \"27aba347-8a24-4cd1-a1dc-1d6b909944a1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.287042 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.287019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27aba347-8a24-4cd1-a1dc-1d6b909944a1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nbk7s\" (UID: \"27aba347-8a24-4cd1-a1dc-1d6b909944a1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.295394 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.295367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwcp\" (UniqueName: \"kubernetes.io/projected/27aba347-8a24-4cd1-a1dc-1d6b909944a1-kube-api-access-njwcp\") pod \"openshift-lws-operator-fd8ccff4c-nbk7s\" (UID: \"27aba347-8a24-4cd1-a1dc-1d6b909944a1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.464405 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.464331 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" Jul 24 22:56:43.581962 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:43.581930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s"] Jul 24 22:56:43.584956 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:56:43.584930 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27aba347_8a24_4cd1_a1dc_1d6b909944a1.slice/crio-3dbb720fbf4322d6bb874eef4d451521c23cf5e51b82f0c0adb6ce5509695ca3 WatchSource:0}: Error finding container 3dbb720fbf4322d6bb874eef4d451521c23cf5e51b82f0c0adb6ce5509695ca3: Status 404 returned error can't find the container with id 3dbb720fbf4322d6bb874eef4d451521c23cf5e51b82f0c0adb6ce5509695ca3 Jul 24 22:56:44.392264 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:44.392215 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" event={"ID":"27aba347-8a24-4cd1-a1dc-1d6b909944a1","Type":"ContainerStarted","Data":"3dbb720fbf4322d6bb874eef4d451521c23cf5e51b82f0c0adb6ce5509695ca3"} Jul 24 22:56:46.401067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:46.400964 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" event={"ID":"27aba347-8a24-4cd1-a1dc-1d6b909944a1","Type":"ContainerStarted","Data":"5c23b603044958f834c0679a7c4ea0dedaf16f40e32c92eabac00874701aa975"} Jul 24 22:56:46.419635 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:56:46.419590 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nbk7s" podStartSLOduration=0.897441175 podStartE2EDuration="3.419575012s" totalImagesPullingTime="2.522133837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:46.418960406 +0000 UTC m=+265.693929696" watchObservedRunningTime="2026-07-24 22:56:46.419575012 +0000 UTC m=+265.694544107" Jul 24 22:57:03.789910 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.789873 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p"] Jul 24 22:57:03.793341 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.793318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:03.796455 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.796431 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 22:57:03.796564 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.796498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 22:57:03.796564 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.796528 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 22:57:03.796651 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.796580 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7r678\"" Jul 24 22:57:03.796882 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.796864 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 22:57:03.811724 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.811704 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p"] Jul 24 22:57:03.935310 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.935278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5c955d8-5c8a-42bf-baad-bfad7ef79702-manifests\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:03.935310 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.935312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhr9\" (UniqueName: \"kubernetes.io/projected/c5c955d8-5c8a-42bf-baad-bfad7ef79702-kube-api-access-vdhr9\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:03.935498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.935333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5c955d8-5c8a-42bf-baad-bfad7ef79702-webhook-cert\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:03.935498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.935389 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5c955d8-5c8a-42bf-baad-bfad7ef79702-apiservice-cert\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:03.935498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:03.935445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5c955d8-5c8a-42bf-baad-bfad7ef79702-tmp\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.036694 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.036645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5c955d8-5c8a-42bf-baad-bfad7ef79702-apiservice-cert\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.036807 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.036714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5c955d8-5c8a-42bf-baad-bfad7ef79702-tmp\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.036807 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.036764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5c955d8-5c8a-42bf-baad-bfad7ef79702-manifests\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.036807 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.036793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhr9\" (UniqueName: \"kubernetes.io/projected/c5c955d8-5c8a-42bf-baad-bfad7ef79702-kube-api-access-vdhr9\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.036956 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.036818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5c955d8-5c8a-42bf-baad-bfad7ef79702-webhook-cert\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.037237 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.037203 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5c955d8-5c8a-42bf-baad-bfad7ef79702-manifests\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.038939 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.038904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5c955d8-5c8a-42bf-baad-bfad7ef79702-tmp\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.039107 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.039090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5c955d8-5c8a-42bf-baad-bfad7ef79702-apiservice-cert\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.039168 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.039127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5c955d8-5c8a-42bf-baad-bfad7ef79702-webhook-cert\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.045350 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.045291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhr9\" (UniqueName: \"kubernetes.io/projected/c5c955d8-5c8a-42bf-baad-bfad7ef79702-kube-api-access-vdhr9\") pod \"opendatahub-operator-controller-manager-777c88b5cf-2ss6p\" (UID: \"c5c955d8-5c8a-42bf-baad-bfad7ef79702\") " pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.103447 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.103414 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:04.230304 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.230281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p"] Jul 24 22:57:04.232973 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:57:04.232943 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c955d8_5c8a_42bf_baad_bfad7ef79702.slice/crio-8d0e178c259b0f2c135d3078cefe3f4489439766bbf3c02114eac173f0dfe1df WatchSource:0}: Error finding container 8d0e178c259b0f2c135d3078cefe3f4489439766bbf3c02114eac173f0dfe1df: Status 404 returned error can't find the container with id 8d0e178c259b0f2c135d3078cefe3f4489439766bbf3c02114eac173f0dfe1df Jul 24 22:57:04.466280 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:04.466246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" event={"ID":"c5c955d8-5c8a-42bf-baad-bfad7ef79702","Type":"ContainerStarted","Data":"8d0e178c259b0f2c135d3078cefe3f4489439766bbf3c02114eac173f0dfe1df"} Jul 24 22:57:07.479867 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:07.479788 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" event={"ID":"c5c955d8-5c8a-42bf-baad-bfad7ef79702","Type":"ContainerStarted","Data":"683498b7969932c375704d92e61adac13be59d271b1e9030c8339ce92b9aa597"} Jul 24 22:57:08.484876 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:08.484784 2562 generic.go:356] "Generic (PLEG): container finished" podID="c5c955d8-5c8a-42bf-baad-bfad7ef79702" containerID="683498b7969932c375704d92e61adac13be59d271b1e9030c8339ce92b9aa597" exitCode=0 Jul 24 22:57:08.484876 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:08.484830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" event={"ID":"c5c955d8-5c8a-42bf-baad-bfad7ef79702","Type":"ContainerDied","Data":"683498b7969932c375704d92e61adac13be59d271b1e9030c8339ce92b9aa597"} Jul 24 22:57:09.490229 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:09.490191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" event={"ID":"c5c955d8-5c8a-42bf-baad-bfad7ef79702","Type":"ContainerStarted","Data":"7ef00111b6a0cf4b16d13963dcb2afe8a1911b830803d7bb19d185b3817285a5"} Jul 24 22:57:09.490629 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:09.490322 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:09.536427 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:09.536371 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" podStartSLOduration=2.242608854 podStartE2EDuration="6.536351819s" totalImagesPullingTime="3.292759815s" totalInitContainerRuntime="1.00098315s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:57:09.534425665 +0000 UTC m=+288.809394767" watchObservedRunningTime="2026-07-24 22:57:09.536351819 +0000 UTC m=+288.811320918" Jul 24 22:57:20.496211 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:20.496181 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-777c88b5cf-2ss6p" Jul 24 22:57:21.104466 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.104437 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:57:21.107707 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.107658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 22:57:21.112125 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.112104 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 22:57:21.860692 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.860644 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk"] Jul 24 22:57:21.879005 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.878983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk"] Jul 24 22:57:21.879120 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.879071 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:21.882888 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.882862 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 22:57:21.882888 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.882865 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 22:57:21.883094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.882865 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 22:57:21.883094 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.882877 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rfbnv\"" Jul 24 22:57:21.971359 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.971328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-cert\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:21.971505 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.971369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-metrics-cert\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:21.971505 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.971392 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-manager-config\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:21.971505 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:21.971489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24q4h\" (UniqueName: \"kubernetes.io/projected/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-kube-api-access-24q4h\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.071867 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.071834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24q4h\" (UniqueName: \"kubernetes.io/projected/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-kube-api-access-24q4h\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.072018 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.071895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-cert\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.072018 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.071944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-metrics-cert\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.072018 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.071978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-manager-config\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.072604 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.072581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-manager-config\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.074274 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.074254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-cert\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.074368 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.074350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-metrics-cert\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.085581 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.085555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24q4h\" (UniqueName: \"kubernetes.io/projected/5cd3fff4-6905-45b7-b74d-c71a5cfa54db-kube-api-access-24q4h\") pod \"lws-controller-manager-6bccc7ffcc-spfnk\" (UID: \"5cd3fff4-6905-45b7-b74d-c71a5cfa54db\") " pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.187701 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.187623 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:22.323385 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.323359 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk"] Jul 24 22:57:22.326071 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:57:22.326037 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd3fff4_6905_45b7_b74d_c71a5cfa54db.slice/crio-549a1c08768cedcf3d8631ed8e432325718090251900026b0065b698d6d03917 WatchSource:0}: Error finding container 549a1c08768cedcf3d8631ed8e432325718090251900026b0065b698d6d03917: Status 404 returned error can't find the container with id 549a1c08768cedcf3d8631ed8e432325718090251900026b0065b698d6d03917 Jul 24 22:57:22.328229 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.328212 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:57:22.534643 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:22.534560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" event={"ID":"5cd3fff4-6905-45b7-b74d-c71a5cfa54db","Type":"ContainerStarted","Data":"549a1c08768cedcf3d8631ed8e432325718090251900026b0065b698d6d03917"} Jul 24 22:57:24.547317 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:24.547269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" event={"ID":"5cd3fff4-6905-45b7-b74d-c71a5cfa54db","Type":"ContainerStarted","Data":"c7ddef9ba260082603a874d48b8d5c06d1043a98e48fd7021c12f080301bdaea"} Jul 24 22:57:24.547760 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:24.547373 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:24.566485 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:24.566425 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" podStartSLOduration=1.917827415 podStartE2EDuration="3.566404933s" totalImagesPullingTime="1.648577518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:57:24.565767877 +0000 UTC m=+303.840736974" watchObservedRunningTime="2026-07-24 22:57:24.566404933 +0000 UTC m=+303.841374029" Jul 24 22:57:30.445455 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.445421 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc"] Jul 24 22:57:30.448360 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.448341 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.450927 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.450897 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-j6cww\"" Jul 24 22:57:30.451032 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.451021 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 22:57:30.462288 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.462268 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc"] Jul 24 22:57:30.638213 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638176 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0c423-8bca-493c-934a-778732a66659-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638213 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638415 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638415 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638415 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64sk\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-kube-api-access-n64sk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638415 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0c423-8bca-493c-934a-778732a66659-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.638550 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.638474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739720 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0c423-8bca-493c-934a-778732a66659-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739720 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739720 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739967 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739967 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64sk\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-kube-api-access-n64sk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739967 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739967 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739967 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0c423-8bca-493c-934a-778732a66659-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.739967 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.739894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.740345 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.740308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.740539 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.740431 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.740653 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.740451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.740653 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.740569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0c423-8bca-493c-934a-778732a66659-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.740870 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.740845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.742156 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.742134 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.742517 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.742498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0c423-8bca-493c-934a-778732a66659-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.748843 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.748815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.749525 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.749505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64sk\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-kube-api-access-n64sk\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.758973 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.758953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:30.826267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.826227 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w"] Jul 24 22:57:30.828660 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.828640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.832015 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.831883 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 22:57:30.832015 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.831910 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 22:57:30.832193 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.832042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 22:57:30.832430 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.832410 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7h7rl\"" Jul 24 22:57:30.855138 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.852637 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w"] Jul 24 22:57:30.915156 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.915126 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc"] Jul 24 22:57:30.919454 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:57:30.919413 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f0c423_8bca_493c_934a_778732a66659.slice/crio-b35a3b2847cfb3c196552f47fc15d55dc03224e3da5c96b9391a52a30274735f WatchSource:0}: Error finding container b35a3b2847cfb3c196552f47fc15d55dc03224e3da5c96b9391a52a30274735f: Status 404 returned error can't find the container with id b35a3b2847cfb3c196552f47fc15d55dc03224e3da5c96b9391a52a30274735f Jul 24 22:57:30.941831 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.941806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.941924 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.941848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.941924 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.941881 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93d50af0-a81a-4a54-b190-797c4c83d412-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.942010 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.941959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d50af0-a81a-4a54-b190-797c4c83d412-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.942010 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.941989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.942134 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.942012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d50af0-a81a-4a54-b190-797c4c83d412-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:30.942134 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:30.942089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xh59\" (UniqueName: \"kubernetes.io/projected/93d50af0-a81a-4a54-b190-797c4c83d412-kube-api-access-8xh59\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043067 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93d50af0-a81a-4a54-b190-797c4c83d412-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043235 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d50af0-a81a-4a54-b190-797c4c83d412-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043235 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043235 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d50af0-a81a-4a54-b190-797c4c83d412-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043235 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xh59\" (UniqueName: \"kubernetes.io/projected/93d50af0-a81a-4a54-b190-797c4c83d412-kube-api-access-8xh59\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043235 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.043841 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.043804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93d50af0-a81a-4a54-b190-797c4c83d412-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.045425 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.045402 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d50af0-a81a-4a54-b190-797c4c83d412-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.045502 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.045441 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.045747 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.045729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.045793 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.045775 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93d50af0-a81a-4a54-b190-797c4c83d412-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.051304 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.051286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d50af0-a81a-4a54-b190-797c4c83d412-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.051697 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.051662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xh59\" (UniqueName: \"kubernetes.io/projected/93d50af0-a81a-4a54-b190-797c4c83d412-kube-api-access-8xh59\") pod \"istiod-openshift-gateway-5677ccb49b-sk55w\" (UID: \"93d50af0-a81a-4a54-b190-797c4c83d412\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.139808 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.139786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:31.300809 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.300786 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w"] Jul 24 22:57:31.301342 ip-10-0-143-241 kubenswrapper[2562]: W0724 22:57:31.301319 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d50af0_a81a_4a54_b190_797c4c83d412.slice/crio-7048cc7d8d92a4b97d1d84153e865e6896bd1063805b44b130821c9f0884e072 WatchSource:0}: Error finding container 7048cc7d8d92a4b97d1d84153e865e6896bd1063805b44b130821c9f0884e072: Status 404 returned error can't find the container with id 7048cc7d8d92a4b97d1d84153e865e6896bd1063805b44b130821c9f0884e072 Jul 24 22:57:31.578460 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.578372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" event={"ID":"93d50af0-a81a-4a54-b190-797c4c83d412","Type":"ContainerStarted","Data":"7048cc7d8d92a4b97d1d84153e865e6896bd1063805b44b130821c9f0884e072"} Jul 24 22:57:31.579462 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:31.579437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" event={"ID":"75f0c423-8bca-493c-934a-778732a66659","Type":"ContainerStarted","Data":"b35a3b2847cfb3c196552f47fc15d55dc03224e3da5c96b9391a52a30274735f"} Jul 24 22:57:35.019722 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.019656 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 22:57:35.019722 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.019695 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 22:57:35.020047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.019763 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 22:57:35.020047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.019776 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 22:57:35.020047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.019813 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 22:57:35.554874 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.554838 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bccc7ffcc-spfnk" Jul 24 22:57:35.598357 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.598318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" event={"ID":"75f0c423-8bca-493c-934a-778732a66659","Type":"ContainerStarted","Data":"75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272"} Jul 24 22:57:35.600502 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.600471 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" event={"ID":"93d50af0-a81a-4a54-b190-797c4c83d412","Type":"ContainerStarted","Data":"0992eb647e3006f0aacb89a1924894be3912775d452075da812e5d09c88ea0af"} Jul 24 22:57:35.600663 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.600565 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:35.626029 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.625971 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" podStartSLOduration=1.5281807889999999 podStartE2EDuration="5.625948536s" totalImagesPullingTime="4.097767747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:57:35.624932596 +0000 UTC m=+314.899901693" watchObservedRunningTime="2026-07-24 22:57:35.625948536 +0000 UTC m=+314.900917632" Jul 24 22:57:35.648517 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.648461 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" podStartSLOduration=1.932287 podStartE2EDuration="5.648444968s" totalImagesPullingTime="3.716157968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:57:35.647530493 +0000 UTC m=+314.922499589" watchObservedRunningTime="2026-07-24 22:57:35.648444968 +0000 UTC m=+314.923414062" Jul 24 22:57:35.759350 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:35.759324 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:36.260218 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:36.260182 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc"] Jul 24 22:57:36.607045 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:36.607016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-sk55w" Jul 24 22:57:36.760520 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:36.760482 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Jul 24 22:57:36.760662 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:36.760554 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" podUID="75f0c423-8bca-493c-934a-778732a66659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Jul 24 22:57:37.609390 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:37.609348 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" podUID="75f0c423-8bca-493c-934a-778732a66659" containerName="istio-proxy" containerID="cri-o://75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272" gracePeriod=30 Jul 24 22:57:42.848932 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.848906 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:42.939013 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.938925 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-socket\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939013 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.938971 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0c423-8bca-493c-934a-778732a66659-istio-podinfo\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939013 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939009 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-credential-socket\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939268 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939030 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0c423-8bca-493c-934a-778732a66659-istiod-ca-cert\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939268 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939044 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-envoy\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939268 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939086 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-data\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939268 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939222 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-istio-token\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939268 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939245 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:57:42.939498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939257 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-certs\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939347 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n64sk\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-kube-api-access-n64sk\") pod \"75f0c423-8bca-493c-934a-778732a66659\" (UID: \"75f0c423-8bca-493c-934a-778732a66659\") " Jul 24 22:57:42.939498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939401 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:57:42.939498 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939431 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:57:42.939709 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939535 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0c423-8bca-493c-934a-778732a66659-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:57:42.939709 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939630 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:42.939709 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939647 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:42.939709 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939661 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:42.939709 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939693 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0c423-8bca-493c-934a-778732a66659-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:42.939960 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.939797 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-data" (OuterVolumeSpecName: "istio-data") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:57:42.942204 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.942177 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-istio-token" (OuterVolumeSpecName: "istio-token") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:57:42.942204 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.942183 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75f0c423-8bca-493c-934a-778732a66659-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 22:57:42.942204 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.942190 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-kube-api-access-n64sk" (OuterVolumeSpecName: "kube-api-access-n64sk") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "kube-api-access-n64sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:57:42.942390 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:42.942174 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75f0c423-8bca-493c-934a-778732a66659" (UID: "75f0c423-8bca-493c-934a-778732a66659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:57:43.040047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.040014 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0c423-8bca-493c-934a-778732a66659-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:43.040047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.040039 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:43.040047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.040049 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0c423-8bca-493c-934a-778732a66659-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:43.040047 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.040056 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:43.040278 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.040065 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64sk\" (UniqueName: \"kubernetes.io/projected/75f0c423-8bca-493c-934a-778732a66659-kube-api-access-n64sk\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 22:57:43.632859 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.632828 2562 generic.go:356] "Generic (PLEG): container finished" podID="75f0c423-8bca-493c-934a-778732a66659" containerID="75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272" exitCode=0 Jul 24 22:57:43.633012 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.632898 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" event={"ID":"75f0c423-8bca-493c-934a-778732a66659","Type":"ContainerDied","Data":"75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272"} Jul 24 22:57:43.633012 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.632948 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" event={"ID":"75f0c423-8bca-493c-934a-778732a66659","Type":"ContainerDied","Data":"b35a3b2847cfb3c196552f47fc15d55dc03224e3da5c96b9391a52a30274735f"} Jul 24 22:57:43.633012 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.632947 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc" Jul 24 22:57:43.633012 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.632965 2562 scope.go:117] "RemoveContainer" containerID="75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272" Jul 24 22:57:43.641579 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.641556 2562 scope.go:117] "RemoveContainer" containerID="75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272" Jul 24 22:57:43.641873 ip-10-0-143-241 kubenswrapper[2562]: E0724 22:57:43.641845 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272\": container with ID starting with 75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272 not found: ID does not exist" containerID="75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272" Jul 24 22:57:43.641957 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.641884 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272"} err="failed to get container status \"75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272\": rpc error: code = NotFound desc = could not find container \"75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272\": container with ID starting with 75ebaf511db218f4cdbcff977d90ec97a5bfcba22c05eb75217fc04035a6f272 not found: ID does not exist" Jul 24 22:57:43.651375 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.651351 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc"] Jul 24 22:57:43.657987 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:43.657964 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcpkzfc"] Jul 24 22:57:45.299267 ip-10-0-143-241 kubenswrapper[2562]: I0724 22:57:45.299225 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f0c423-8bca-493c-934a-778732a66659" path="/var/lib/kubelet/pods/75f0c423-8bca-493c-934a-778732a66659/volumes" Jul 24 23:00:00.818213 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.818179 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7"] Jul 24 23:00:00.818619 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.818471 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f0c423-8bca-493c-934a-778732a66659" containerName="istio-proxy" Jul 24 23:00:00.818619 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.818482 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f0c423-8bca-493c-934a-778732a66659" containerName="istio-proxy" Jul 24 23:00:00.818619 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.818528 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f0c423-8bca-493c-934a-778732a66659" containerName="istio-proxy" Jul 24 23:00:00.821296 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.821276 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.824575 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.824555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rtxpr\"" Jul 24 23:00:00.837660 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.837633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7"] Jul 24 23:00:00.944544 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.944767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.944767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.944767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.944767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.944767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.945030 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944785 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48xj\" (UniqueName: \"kubernetes.io/projected/eb1c31c7-d8f4-452f-b987-7ecbddadb567-kube-api-access-q48xj\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.945030 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:00.945030 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:00.944848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046112 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48xj\" (UniqueName: \"kubernetes.io/projected/eb1c31c7-d8f4-452f-b987-7ecbddadb567-kube-api-access-q48xj\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046648 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046757 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046733 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.046955 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046929 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.047081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.046978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.047081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.047003 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.047174 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.047024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.048612 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.048583 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.048749 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.048729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.054415 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.054394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1c31c7-d8f4-452f-b987-7ecbddadb567-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.054569 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.054551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48xj\" (UniqueName: \"kubernetes.io/projected/eb1c31c7-d8f4-452f-b987-7ecbddadb567-kube-api-access-q48xj\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4fpq7\" (UID: \"eb1c31c7-d8f4-452f-b987-7ecbddadb567\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.130917 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.130889 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:01.257399 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.257364 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7"] Jul 24 23:00:01.260064 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:00:01.260033 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1c31c7_d8f4_452f_b987_7ecbddadb567.slice/crio-c5ae0fa20c5abda2619a518bf8383a600533100e009cb85013ee56303b6c30bc WatchSource:0}: Error finding container c5ae0fa20c5abda2619a518bf8383a600533100e009cb85013ee56303b6c30bc: Status 404 returned error can't find the container with id c5ae0fa20c5abda2619a518bf8383a600533100e009cb85013ee56303b6c30bc Jul 24 23:00:01.262079 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.262044 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:00:01.262155 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.262127 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:00:01.262205 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:01.262177 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:00:02.165292 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:02.165255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" event={"ID":"eb1c31c7-d8f4-452f-b987-7ecbddadb567","Type":"ContainerStarted","Data":"09cf4f751a2d5c64f115090c606146c2c7b1995c17058e2de6ccce5eefb87857"} Jul 24 23:00:02.165660 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:02.165307 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" event={"ID":"eb1c31c7-d8f4-452f-b987-7ecbddadb567","Type":"ContainerStarted","Data":"c5ae0fa20c5abda2619a518bf8383a600533100e009cb85013ee56303b6c30bc"} Jul 24 23:00:02.188605 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:02.188546 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" podStartSLOduration=2.188526582 podStartE2EDuration="2.188526582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:00:02.187459469 +0000 UTC m=+461.462428564" watchObservedRunningTime="2026-07-24 23:00:02.188526582 +0000 UTC m=+461.463495680" Jul 24 23:00:03.131088 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:03.131049 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:03.135949 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:03.135921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:03.169695 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:03.169636 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:03.170867 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:03.170695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:00:51.493508 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.493473 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57dbb6486d-g57sx"] Jul 24 23:00:51.496541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.496520 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:51.499098 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.499075 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 23:00:51.499214 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.499122 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dq7r7\"" Jul 24 23:00:51.500240 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.500219 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 23:00:51.507101 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.507081 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-g57sx"] Jul 24 23:00:51.523101 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.523078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqpn\" (UniqueName: \"kubernetes.io/projected/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-kube-api-access-stqpn\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:51.523203 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.523111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:51.623487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.623460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:51.623637 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.623561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqpn\" (UniqueName: \"kubernetes.io/projected/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-kube-api-access-stqpn\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:51.623637 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:00:51.623613 2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 23:00:51.623756 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:00:51.623693 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert podName:66c6127c-9f20-4461-97ea-14d3c5ce5ffc nodeName:}" failed. No retries permitted until 2026-07-24 23:00:52.123660139 +0000 UTC m=+511.398629217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert") pod "maas-controller-57dbb6486d-g57sx" (UID: "66c6127c-9f20-4461-97ea-14d3c5ce5ffc") : secret "maas-controller-webhook-cert" not found Jul 24 23:00:51.633295 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:51.633273 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqpn\" (UniqueName: \"kubernetes.io/projected/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-kube-api-access-stqpn\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:52.126916 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:52.126879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:52.129653 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:52.129630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert\") pod \"maas-controller-57dbb6486d-g57sx\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:52.406900 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:52.406812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:52.532856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:52.532826 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-g57sx"] Jul 24 23:00:52.536158 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:00:52.536130 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c6127c_9f20_4461_97ea_14d3c5ce5ffc.slice/crio-9fd67bcebbd719d7a70bc0577f234254f0cd0f33ab11e80dd33693d3ca810a51 WatchSource:0}: Error finding container 9fd67bcebbd719d7a70bc0577f234254f0cd0f33ab11e80dd33693d3ca810a51: Status 404 returned error can't find the container with id 9fd67bcebbd719d7a70bc0577f234254f0cd0f33ab11e80dd33693d3ca810a51 Jul 24 23:00:53.356866 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:53.356830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-g57sx" event={"ID":"66c6127c-9f20-4461-97ea-14d3c5ce5ffc","Type":"ContainerStarted","Data":"9fd67bcebbd719d7a70bc0577f234254f0cd0f33ab11e80dd33693d3ca810a51"} Jul 24 23:00:56.371352 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:56.371316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-g57sx" event={"ID":"66c6127c-9f20-4461-97ea-14d3c5ce5ffc","Type":"ContainerStarted","Data":"fe55a93c32afd1a045def7ed3fd8003b64e1171faf3f16bb37e47691ce2f5a79"} Jul 24 23:00:56.371725 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:56.371446 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:00:56.389296 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:00:56.389247 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57dbb6486d-g57sx" podStartSLOduration=1.96001648 podStartE2EDuration="5.389230221s" totalImagesPullingTime="3.429213741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:00:56.388452149 +0000 UTC m=+515.663421243" watchObservedRunningTime="2026-07-24 23:00:56.389230221 +0000 UTC m=+515.664199315" Jul 24 23:01:07.377227 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:07.377197 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:01:29.124710 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.124653 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ndrmz"] Jul 24 23:01:29.132988 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.132959 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.135896 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.135872 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 23:01:29.136070 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.135921 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pdxdc\"" Jul 24 23:01:29.139171 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.139149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ndrmz"] Jul 24 23:01:29.227757 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.227726 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzp2x\" (UniqueName: \"kubernetes.io/projected/d4090740-da29-4dad-b5b7-8a587436717d-kube-api-access-vzp2x\") pod \"payload-processing-5cc6cd444b-ndrmz\" (UID: \"d4090740-da29-4dad-b5b7-8a587436717d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.227909 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.227768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4090740-da29-4dad-b5b7-8a587436717d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ndrmz\" (UID: \"d4090740-da29-4dad-b5b7-8a587436717d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.328567 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.328534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzp2x\" (UniqueName: \"kubernetes.io/projected/d4090740-da29-4dad-b5b7-8a587436717d-kube-api-access-vzp2x\") pod \"payload-processing-5cc6cd444b-ndrmz\" (UID: \"d4090740-da29-4dad-b5b7-8a587436717d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.328740 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.328572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4090740-da29-4dad-b5b7-8a587436717d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ndrmz\" (UID: \"d4090740-da29-4dad-b5b7-8a587436717d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.329218 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.329199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4090740-da29-4dad-b5b7-8a587436717d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-ndrmz\" (UID: \"d4090740-da29-4dad-b5b7-8a587436717d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.337521 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.337497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzp2x\" (UniqueName: \"kubernetes.io/projected/d4090740-da29-4dad-b5b7-8a587436717d-kube-api-access-vzp2x\") pod \"payload-processing-5cc6cd444b-ndrmz\" (UID: \"d4090740-da29-4dad-b5b7-8a587436717d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.443990 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.443927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:29.573841 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:29.573816 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-ndrmz"] Jul 24 23:01:29.576301 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:01:29.576276 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4090740_da29_4dad_b5b7_8a587436717d.slice/crio-fbc6b611285c2ad9f01845b31c139784c0a217f36812e94769cbd1f8748412d6 WatchSource:0}: Error finding container fbc6b611285c2ad9f01845b31c139784c0a217f36812e94769cbd1f8748412d6: Status 404 returned error can't find the container with id fbc6b611285c2ad9f01845b31c139784c0a217f36812e94769cbd1f8748412d6 Jul 24 23:01:30.509340 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:30.509297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" event={"ID":"d4090740-da29-4dad-b5b7-8a587436717d","Type":"ContainerStarted","Data":"fbc6b611285c2ad9f01845b31c139784c0a217f36812e94769cbd1f8748412d6"} Jul 24 23:01:32.519517 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:32.519470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" event={"ID":"d4090740-da29-4dad-b5b7-8a587436717d","Type":"ContainerStarted","Data":"b2aa552bc7e9deda9aa1e5c9955beb3910a2c98b3753015c159ccefdd737c27e"} Jul 24 23:01:32.519885 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:32.519610 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:37.542113 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.542077 2562 generic.go:356] "Generic (PLEG): container finished" podID="eb1c31c7-d8f4-452f-b987-7ecbddadb567" containerID="09cf4f751a2d5c64f115090c606146c2c7b1995c17058e2de6ccce5eefb87857" exitCode=0 Jul 24 23:01:37.542525 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.542155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" event={"ID":"eb1c31c7-d8f4-452f-b987-7ecbddadb567","Type":"ContainerDied","Data":"09cf4f751a2d5c64f115090c606146c2c7b1995c17058e2de6ccce5eefb87857"} Jul 24 23:01:37.542525 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.542496 2562 scope.go:117] "RemoveContainer" containerID="09cf4f751a2d5c64f115090c606146c2c7b1995c17058e2de6ccce5eefb87857" Jul 24 23:01:37.543438 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.543409 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:01:37.543496 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.543479 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:01:37.543533 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.543519 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:01:37.563547 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:37.563505 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" podStartSLOduration=5.68637015 podStartE2EDuration="8.563487816s" totalImagesPullingTime="2.877117666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:01:32.536997836 +0000 UTC m=+551.811966942" watchObservedRunningTime="2026-07-24 23:01:37.563487816 +0000 UTC m=+556.838456910" Jul 24 23:01:38.553074 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:38.553042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" event={"ID":"eb1c31c7-d8f4-452f-b987-7ecbddadb567","Type":"ContainerStarted","Data":"83e16840dff371addc7f32401e7f1132a400d8affbaf106f5460c737ee984e0c"} Jul 24 23:01:39.131951 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:39.131856 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:01:39.132150 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:39.132110 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4fpq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 24 23:01:39.132227 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:39.132162 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" podUID="eb1c31c7-d8f4-452f-b987-7ecbddadb567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 24 23:01:40.131790 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:40.131744 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4fpq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 24 23:01:40.132257 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:40.131813 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" podUID="eb1c31c7-d8f4-452f-b987-7ecbddadb567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 24 23:01:41.132344 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:41.132299 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4fpq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 24 23:01:41.132864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:41.132370 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" podUID="eb1c31c7-d8f4-452f-b987-7ecbddadb567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 24 23:01:42.132243 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:42.132202 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4fpq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body= Jul 24 23:01:42.132538 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:42.132509 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" podUID="eb1c31c7-d8f4-452f-b987-7ecbddadb567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" Jul 24 23:01:43.135410 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:43.135377 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:01:43.135855 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:43.135655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:01:43.136378 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:43.136363 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4fpq7" Jul 24 23:01:43.526534 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:43.526448 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-ndrmz" Jul 24 23:01:46.484164 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.484127 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r"] Jul 24 23:01:46.487488 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.487470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.490117 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.490098 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 23:01:46.490117 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.490107 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 23:01:46.492111 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.492095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q7cqg\"" Jul 24 23:01:46.492169 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.492095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 23:01:46.498092 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.498067 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r"] Jul 24 23:01:46.586168 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.586136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.586330 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.586175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jblp\" (UniqueName: \"kubernetes.io/projected/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-kube-api-access-9jblp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.586330 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.586253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.586330 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.586295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.586330 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.586326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.586507 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.586390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687207 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jblp\" (UniqueName: \"kubernetes.io/projected/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-kube-api-access-9jblp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687295 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687831 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687777 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.687891 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.687875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.689527 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.689502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.689710 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.689694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.694718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.694693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jblp\" (UniqueName: \"kubernetes.io/projected/5ffad835-f0f2-465e-ab4d-07e4d6037ba5-kube-api-access-9jblp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r\" (UID: \"5ffad835-f0f2-465e-ab4d-07e4d6037ba5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.798099 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.798022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:46.923294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:46.923257 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r"] Jul 24 23:01:46.925462 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:01:46.925432 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffad835_f0f2_465e_ab4d_07e4d6037ba5.slice/crio-1061100e760096c5af03f658810cee9e727020fb56701f5f3222dfa51b2fcd6b WatchSource:0}: Error finding container 1061100e760096c5af03f658810cee9e727020fb56701f5f3222dfa51b2fcd6b: Status 404 returned error can't find the container with id 1061100e760096c5af03f658810cee9e727020fb56701f5f3222dfa51b2fcd6b Jul 24 23:01:47.596334 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:47.596295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" event={"ID":"5ffad835-f0f2-465e-ab4d-07e4d6037ba5","Type":"ContainerStarted","Data":"1061100e760096c5af03f658810cee9e727020fb56701f5f3222dfa51b2fcd6b"} Jul 24 23:01:48.602063 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:48.602028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" event={"ID":"5ffad835-f0f2-465e-ab4d-07e4d6037ba5","Type":"ContainerStarted","Data":"0df6c9135c11d9e082c58791ece7bfa4abc4b864aabfeb84ebc2147f213ddb76"} Jul 24 23:01:56.798488 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:56.798444 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:56.798909 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:56.798517 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:56.810881 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:56.810859 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:01:56.830877 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:56.830800 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" podStartSLOduration=9.720592011 podStartE2EDuration="10.830784454s" totalImagesPullingTime="1.110192443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:01:48.61885614 +0000 UTC m=+567.893825235" watchObservedRunningTime="2026-07-24 23:01:56.830784454 +0000 UTC m=+576.105753549" Jul 24 23:01:57.649309 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:01:57.649278 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r" Jul 24 23:02:16.590031 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.589986 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp"] Jul 24 23:02:16.593609 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.593591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.596081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.596055 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 23:02:16.600685 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.600647 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp"] Jul 24 23:02:16.725294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.725251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.725294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.725299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.725518 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.725337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.725518 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.725421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.725518 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.725460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfzt\" (UniqueName: \"kubernetes.io/projected/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-kube-api-access-wlfzt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.725616 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.725535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826261 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfzt\" (UniqueName: \"kubernetes.io/projected/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-kube-api-access-wlfzt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826787 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826874 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.826874 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.826839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.828582 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.828562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.828778 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.828761 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.835326 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.835305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfzt\" (UniqueName: \"kubernetes.io/projected/c106af9b-3830-4b25-a32e-2aa2f23c6b5b-kube-api-access-wlfzt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp\" (UID: \"c106af9b-3830-4b25-a32e-2aa2f23c6b5b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:16.905360 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:16.905270 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:17.035326 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:17.035298 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp"] Jul 24 23:02:17.037708 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:02:17.037661 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc106af9b_3830_4b25_a32e_2aa2f23c6b5b.slice/crio-9c103ffc174cb574ecaab57817869a08e9ebc01d71eabc2e59c9e590c119c906 WatchSource:0}: Error finding container 9c103ffc174cb574ecaab57817869a08e9ebc01d71eabc2e59c9e590c119c906: Status 404 returned error can't find the container with id 9c103ffc174cb574ecaab57817869a08e9ebc01d71eabc2e59c9e590c119c906 Jul 24 23:02:17.722348 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:17.722312 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" event={"ID":"c106af9b-3830-4b25-a32e-2aa2f23c6b5b","Type":"ContainerStarted","Data":"5da8ab94f297aa328c4562bb51eccc3e99ac0634bc8e8e636fe238925745d6ef"} Jul 24 23:02:17.722755 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:17.722365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" event={"ID":"c106af9b-3830-4b25-a32e-2aa2f23c6b5b","Type":"ContainerStarted","Data":"9c103ffc174cb574ecaab57817869a08e9ebc01d71eabc2e59c9e590c119c906"} Jul 24 23:02:17.739549 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:17.739499 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" podStartSLOduration=1.5657580599999998 podStartE2EDuration="1.73948358s" totalImagesPullingTime="173.72552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:02:17.738709107 +0000 UTC m=+597.013678204" watchObservedRunningTime="2026-07-24 23:02:17.73948358 +0000 UTC m=+597.014452675" Jul 24 23:02:21.131653 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:21.131628 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:02:21.131653 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:21.131638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:02:26.905880 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:26.905846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:26.906347 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:26.905912 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:26.918185 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:26.918164 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:02:27.776558 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:02:27.776531 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp" Jul 24 23:04:21.051539 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.051502 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-g57sx"] Jul 24 23:04:21.052091 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.051715 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57dbb6486d-g57sx" podUID="66c6127c-9f20-4461-97ea-14d3c5ce5ffc" containerName="manager" containerID="cri-o://fe55a93c32afd1a045def7ed3fd8003b64e1171faf3f16bb37e47691ce2f5a79" gracePeriod=30 Jul 24 23:04:21.239236 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.239197 2562 generic.go:356] "Generic (PLEG): container finished" podID="66c6127c-9f20-4461-97ea-14d3c5ce5ffc" containerID="fe55a93c32afd1a045def7ed3fd8003b64e1171faf3f16bb37e47691ce2f5a79" exitCode=0 Jul 24 23:04:21.239380 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.239266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-g57sx" event={"ID":"66c6127c-9f20-4461-97ea-14d3c5ce5ffc","Type":"ContainerDied","Data":"fe55a93c32afd1a045def7ed3fd8003b64e1171faf3f16bb37e47691ce2f5a79"} Jul 24 23:04:21.301366 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.301328 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:04:21.438888 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.438857 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stqpn\" (UniqueName: \"kubernetes.io/projected/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-kube-api-access-stqpn\") pod \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " Jul 24 23:04:21.439092 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.438954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert\") pod \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\" (UID: \"66c6127c-9f20-4461-97ea-14d3c5ce5ffc\") " Jul 24 23:04:21.441101 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.441068 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-kube-api-access-stqpn" (OuterVolumeSpecName: "kube-api-access-stqpn") pod "66c6127c-9f20-4461-97ea-14d3c5ce5ffc" (UID: "66c6127c-9f20-4461-97ea-14d3c5ce5ffc"). InnerVolumeSpecName "kube-api-access-stqpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:04:21.441212 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.441118 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "66c6127c-9f20-4461-97ea-14d3c5ce5ffc" (UID: "66c6127c-9f20-4461-97ea-14d3c5ce5ffc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:04:21.540487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.540448 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-webhook-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:04:21.540487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:21.540482 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqpn\" (UniqueName: \"kubernetes.io/projected/66c6127c-9f20-4461-97ea-14d3c5ce5ffc-kube-api-access-stqpn\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:04:22.244791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:22.244752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-g57sx" event={"ID":"66c6127c-9f20-4461-97ea-14d3c5ce5ffc","Type":"ContainerDied","Data":"9fd67bcebbd719d7a70bc0577f234254f0cd0f33ab11e80dd33693d3ca810a51"} Jul 24 23:04:22.245269 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:22.244798 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-g57sx" Jul 24 23:04:22.245269 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:22.244813 2562 scope.go:117] "RemoveContainer" containerID="fe55a93c32afd1a045def7ed3fd8003b64e1171faf3f16bb37e47691ce2f5a79" Jul 24 23:04:22.265747 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:22.265720 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-g57sx"] Jul 24 23:04:22.268379 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:22.268351 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-g57sx"] Jul 24 23:04:23.298787 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:04:23.298754 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c6127c-9f20-4461-97ea-14d3c5ce5ffc" path="/var/lib/kubelet/pods/66c6127c-9f20-4461-97ea-14d3c5ce5ffc/volumes" Jul 24 23:05:23.075004 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.074973 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57dbb6486d-kjn7w"] Jul 24 23:05:23.075381 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.075233 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c6127c-9f20-4461-97ea-14d3c5ce5ffc" containerName="manager" Jul 24 23:05:23.075381 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.075244 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c6127c-9f20-4461-97ea-14d3c5ce5ffc" containerName="manager" Jul 24 23:05:23.075381 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.075299 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c6127c-9f20-4461-97ea-14d3c5ce5ffc" containerName="manager" Jul 24 23:05:23.079218 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.079196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.082880 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.082859 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 23:05:23.082880 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.082873 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 23:05:23.083034 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.082859 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dq7r7\"" Jul 24 23:05:23.089268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.089246 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-kjn7w"] Jul 24 23:05:23.155439 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.155403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-webhook-cert\") pod \"maas-controller-57dbb6486d-kjn7w\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.155576 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.155473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8bs\" (UniqueName: \"kubernetes.io/projected/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-kube-api-access-vr8bs\") pod \"maas-controller-57dbb6486d-kjn7w\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.256218 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.256174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8bs\" (UniqueName: \"kubernetes.io/projected/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-kube-api-access-vr8bs\") pod \"maas-controller-57dbb6486d-kjn7w\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.256351 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.256240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-webhook-cert\") pod \"maas-controller-57dbb6486d-kjn7w\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.258601 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.258573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-webhook-cert\") pod \"maas-controller-57dbb6486d-kjn7w\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.264065 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.264040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8bs\" (UniqueName: \"kubernetes.io/projected/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-kube-api-access-vr8bs\") pod \"maas-controller-57dbb6486d-kjn7w\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.391149 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.391110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:23.526020 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.525990 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-kjn7w"] Jul 24 23:05:23.529850 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:05:23.529828 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3e76a5_fc97_4e1e_afc2_52259e6bb692.slice/crio-633a6072f891609e7d831d19449f85faf202fadc89e26efcadb1131f41df3dd2 WatchSource:0}: Error finding container 633a6072f891609e7d831d19449f85faf202fadc89e26efcadb1131f41df3dd2: Status 404 returned error can't find the container with id 633a6072f891609e7d831d19449f85faf202fadc89e26efcadb1131f41df3dd2 Jul 24 23:05:23.531183 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:23.531162 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:05:24.496586 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:24.496538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" event={"ID":"1c3e76a5-fc97-4e1e-afc2-52259e6bb692","Type":"ContainerStarted","Data":"e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572"} Jul 24 23:05:24.497107 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:24.496646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" event={"ID":"1c3e76a5-fc97-4e1e-afc2-52259e6bb692","Type":"ContainerStarted","Data":"633a6072f891609e7d831d19449f85faf202fadc89e26efcadb1131f41df3dd2"} Jul 24 23:05:24.497107 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:24.496701 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:24.517833 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:24.517172 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" podStartSLOduration=1.067450899 podStartE2EDuration="1.517152561s" totalImagesPullingTime="449.701662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:05:24.516869218 +0000 UTC m=+783.791838317" watchObservedRunningTime="2026-07-24 23:05:24.517152561 +0000 UTC m=+783.792121657" Jul 24 23:05:35.502698 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:35.502643 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:40.161374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.161335 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-kjn7w"] Jul 24 23:05:40.161802 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.161541 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" podUID="1c3e76a5-fc97-4e1e-afc2-52259e6bb692" containerName="manager" containerID="cri-o://e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572" gracePeriod=30 Jul 24 23:05:40.400553 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.400530 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:40.495660 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.495590 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr8bs\" (UniqueName: \"kubernetes.io/projected/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-kube-api-access-vr8bs\") pod \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " Jul 24 23:05:40.495816 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.495721 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-webhook-cert\") pod \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\" (UID: \"1c3e76a5-fc97-4e1e-afc2-52259e6bb692\") " Jul 24 23:05:40.497565 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.497534 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1c3e76a5-fc97-4e1e-afc2-52259e6bb692" (UID: "1c3e76a5-fc97-4e1e-afc2-52259e6bb692"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:05:40.497696 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.497544 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-kube-api-access-vr8bs" (OuterVolumeSpecName: "kube-api-access-vr8bs") pod "1c3e76a5-fc97-4e1e-afc2-52259e6bb692" (UID: "1c3e76a5-fc97-4e1e-afc2-52259e6bb692"). InnerVolumeSpecName "kube-api-access-vr8bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:40.561976 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.561947 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c3e76a5-fc97-4e1e-afc2-52259e6bb692" containerID="e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572" exitCode=0 Jul 24 23:05:40.562084 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.561986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" event={"ID":"1c3e76a5-fc97-4e1e-afc2-52259e6bb692","Type":"ContainerDied","Data":"e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572"} Jul 24 23:05:40.562084 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.561994 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" Jul 24 23:05:40.562084 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.562023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-kjn7w" event={"ID":"1c3e76a5-fc97-4e1e-afc2-52259e6bb692","Type":"ContainerDied","Data":"633a6072f891609e7d831d19449f85faf202fadc89e26efcadb1131f41df3dd2"} Jul 24 23:05:40.562084 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.562045 2562 scope.go:117] "RemoveContainer" containerID="e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572" Jul 24 23:05:40.570158 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.570138 2562 scope.go:117] "RemoveContainer" containerID="e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572" Jul 24 23:05:40.570423 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:05:40.570404 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572\": container with ID starting with e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572 not found: ID does not exist" containerID="e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572" Jul 24 23:05:40.570491 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.570436 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572"} err="failed to get container status \"e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572\": rpc error: code = NotFound desc = could not find container \"e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572\": container with ID starting with e2e924d9b23c64f11ee4fbfd34a76a17cbb55375f451def264bd86ab8601a572 not found: ID does not exist" Jul 24 23:05:40.582636 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.582617 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-kjn7w"] Jul 24 23:05:40.585702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.585668 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-kjn7w"] Jul 24 23:05:40.596795 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.596778 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-webhook-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:05:40.596863 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:40.596798 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8bs\" (UniqueName: \"kubernetes.io/projected/1c3e76a5-fc97-4e1e-afc2-52259e6bb692-kube-api-access-vr8bs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:05:41.302120 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.302088 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3e76a5-fc97-4e1e-afc2-52259e6bb692" path="/var/lib/kubelet/pods/1c3e76a5-fc97-4e1e-afc2-52259e6bb692/volumes" Jul 24 23:05:41.367871 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.367838 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57dbb6486d-xv9p8"] Jul 24 23:05:41.368153 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.368139 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3e76a5-fc97-4e1e-afc2-52259e6bb692" containerName="manager" Jul 24 23:05:41.368206 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.368156 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3e76a5-fc97-4e1e-afc2-52259e6bb692" containerName="manager" Jul 24 23:05:41.368244 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.368209 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3e76a5-fc97-4e1e-afc2-52259e6bb692" containerName="manager" Jul 24 23:05:41.372888 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.372870 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.375284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.375261 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dq7r7\"" Jul 24 23:05:41.375284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.375263 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 23:05:41.376706 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.376690 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 23:05:41.381716 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.381667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-xv9p8"] Jul 24 23:05:41.402227 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.402204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwjfm\" (UniqueName: \"kubernetes.io/projected/4115be71-ee96-45b7-9278-58cf51bef9c9-kube-api-access-fwjfm\") pod \"maas-controller-57dbb6486d-xv9p8\" (UID: \"4115be71-ee96-45b7-9278-58cf51bef9c9\") " pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.402329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.402268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4115be71-ee96-45b7-9278-58cf51bef9c9-webhook-cert\") pod \"maas-controller-57dbb6486d-xv9p8\" (UID: \"4115be71-ee96-45b7-9278-58cf51bef9c9\") " pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.503609 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.503580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4115be71-ee96-45b7-9278-58cf51bef9c9-webhook-cert\") pod \"maas-controller-57dbb6486d-xv9p8\" (UID: \"4115be71-ee96-45b7-9278-58cf51bef9c9\") " pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.503761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.503656 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwjfm\" (UniqueName: \"kubernetes.io/projected/4115be71-ee96-45b7-9278-58cf51bef9c9-kube-api-access-fwjfm\") pod \"maas-controller-57dbb6486d-xv9p8\" (UID: \"4115be71-ee96-45b7-9278-58cf51bef9c9\") " pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.505958 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.505938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4115be71-ee96-45b7-9278-58cf51bef9c9-webhook-cert\") pod \"maas-controller-57dbb6486d-xv9p8\" (UID: \"4115be71-ee96-45b7-9278-58cf51bef9c9\") " pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.511929 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.511909 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwjfm\" (UniqueName: \"kubernetes.io/projected/4115be71-ee96-45b7-9278-58cf51bef9c9-kube-api-access-fwjfm\") pod \"maas-controller-57dbb6486d-xv9p8\" (UID: \"4115be71-ee96-45b7-9278-58cf51bef9c9\") " pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.683988 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.683953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:41.811292 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:41.811268 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dbb6486d-xv9p8"] Jul 24 23:05:41.813781 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:05:41.813754 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4115be71_ee96_45b7_9278_58cf51bef9c9.slice/crio-3f1f9c53efac0c4e10c7177acc1fbb85f9a2c2214c889fd5e5487bdf263b1c05 WatchSource:0}: Error finding container 3f1f9c53efac0c4e10c7177acc1fbb85f9a2c2214c889fd5e5487bdf263b1c05: Status 404 returned error can't find the container with id 3f1f9c53efac0c4e10c7177acc1fbb85f9a2c2214c889fd5e5487bdf263b1c05 Jul 24 23:05:42.572784 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:42.572694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" event={"ID":"4115be71-ee96-45b7-9278-58cf51bef9c9","Type":"ContainerStarted","Data":"3f6b2be1bcef4e75512f2ccee0ebe1a7382f710278e4b0ede5c23a4525c27e02"} Jul 24 23:05:42.572784 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:42.572751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" event={"ID":"4115be71-ee96-45b7-9278-58cf51bef9c9","Type":"ContainerStarted","Data":"3f1f9c53efac0c4e10c7177acc1fbb85f9a2c2214c889fd5e5487bdf263b1c05"} Jul 24 23:05:42.573154 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:42.572785 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:05:42.592404 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:42.592331 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" podStartSLOduration=1.274814395 podStartE2EDuration="1.59231647s" totalImagesPullingTime="317.502075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:05:42.590102907 +0000 UTC m=+801.865072002" watchObservedRunningTime="2026-07-24 23:05:42.59231647 +0000 UTC m=+801.867285567" Jul 24 23:05:53.578728 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:05:53.578698 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57dbb6486d-xv9p8" Jul 24 23:07:21.153240 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:07:21.153213 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:07:21.155013 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:07:21.154992 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:12:21.182859 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:12:21.182817 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:12:21.185379 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:12:21.185354 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:15:00.149000 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.148921 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8"] Jul 24 23:15:00.151942 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.151927 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" Jul 24 23:15:00.154386 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.154369 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h9zqg\"" Jul 24 23:15:00.159616 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.159580 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8"] Jul 24 23:15:00.162069 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.162049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" Jul 24 23:15:00.286824 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.286800 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8"] Jul 24 23:15:00.289388 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:15:00.289360 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8a366d_f5a5_4437_8939_496a1a1d2bd7.slice/crio-a95253a3c40e47302ec80d1bfc245389cbd8f782b4068ec0fc06ad31fbc5db9e WatchSource:0}: Error finding container a95253a3c40e47302ec80d1bfc245389cbd8f782b4068ec0fc06ad31fbc5db9e: Status 404 returned error can't find the container with id a95253a3c40e47302ec80d1bfc245389cbd8f782b4068ec0fc06ad31fbc5db9e Jul 24 23:15:00.291503 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.291480 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:15:00.840040 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:00.840003 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" event={"ID":"2d8a366d-f5a5-4437-8939-496a1a1d2bd7","Type":"ContainerStarted","Data":"a95253a3c40e47302ec80d1bfc245389cbd8f782b4068ec0fc06ad31fbc5db9e"} Jul 24 23:15:03.854258 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:03.854220 2562 generic.go:356] "Generic (PLEG): container finished" podID="2d8a366d-f5a5-4437-8939-496a1a1d2bd7" containerID="bebb4acaa9b83436bca793696631e2ec338a6016a25f71eb5982458b0dc02378" exitCode=0 Jul 24 23:15:03.854617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:03.854271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" event={"ID":"2d8a366d-f5a5-4437-8939-496a1a1d2bd7","Type":"ContainerDied","Data":"bebb4acaa9b83436bca793696631e2ec338a6016a25f71eb5982458b0dc02378"} Jul 24 23:15:04.983836 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:04.983812 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" Jul 24 23:15:05.863636 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:05.863600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" event={"ID":"2d8a366d-f5a5-4437-8939-496a1a1d2bd7","Type":"ContainerDied","Data":"a95253a3c40e47302ec80d1bfc245389cbd8f782b4068ec0fc06ad31fbc5db9e"} Jul 24 23:15:05.863636 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:05.863628 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-nbhr8" Jul 24 23:15:05.863859 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:15:05.863651 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a95253a3c40e47302ec80d1bfc245389cbd8f782b4068ec0fc06ad31fbc5db9e" Jul 24 23:17:21.205451 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:17:21.205333 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:17:21.209427 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:17:21.208546 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:19:03.288881 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.288849 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr"] Jul 24 23:19:03.289356 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.289174 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8a366d-f5a5-4437-8939-496a1a1d2bd7" containerName="cleanup" Jul 24 23:19:03.289356 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.289185 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8a366d-f5a5-4437-8939-496a1a1d2bd7" containerName="cleanup" Jul 24 23:19:03.289356 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.289240 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8a366d-f5a5-4437-8939-496a1a1d2bd7" containerName="cleanup" Jul 24 23:19:03.291988 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.291970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.294546 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.294516 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a599b375-openshift-default-dockercfg-gnxgc\"" Jul 24 23:19:03.313703 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.313658 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr"] Jul 24 23:19:03.320744 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-certs\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.320862 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll6x\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-kube-api-access-mll6x\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.320862 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-envoy\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.320862 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-istio-token\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.321035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53575bf-3458-4884-9078-9edf6c97d98d-istiod-ca-cert\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.321035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-data\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.321035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.320987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-socket\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.321035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.321008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-credential-socket\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.321035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.321030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53575bf-3458-4884-9078-9edf6c97d98d-istio-podinfo\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422412 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-certs\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll6x\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-kube-api-access-mll6x\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-envoy\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-istio-token\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53575bf-3458-4884-9078-9edf6c97d98d-istiod-ca-cert\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-data\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422762 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-socket\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.422803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-certs\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.423002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-credential-socket\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.423002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.422863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53575bf-3458-4884-9078-9edf6c97d98d-istio-podinfo\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.423108 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.423075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-data\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.423303 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.423271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-credential-socket\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.423303 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.423296 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-socket\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.423470 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.423447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53575bf-3458-4884-9078-9edf6c97d98d-istiod-ca-cert\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.424890 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.424870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-envoy\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.425296 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.425274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53575bf-3458-4884-9078-9edf6c97d98d-istio-podinfo\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.430658 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.430635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-istio-token\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.431064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.431048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll6x\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-kube-api-access-mll6x\") pod \"e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.602770 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.602736 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:03.742123 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.742089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr"] Jul 24 23:19:03.745040 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:19:03.745013 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53575bf_3458_4884_9078_9edf6c97d98d.slice/crio-9b1b38b4abcfadafa8d9bf6fc6105dbec183ba7389297f417a771cea8842af71 WatchSource:0}: Error finding container 9b1b38b4abcfadafa8d9bf6fc6105dbec183ba7389297f417a771cea8842af71: Status 404 returned error can't find the container with id 9b1b38b4abcfadafa8d9bf6fc6105dbec183ba7389297f417a771cea8842af71 Jul 24 23:19:03.747188 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.747159 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:03.747265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.747220 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:03.747265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.747250 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:03.838035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.838003 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" event={"ID":"f53575bf-3458-4884-9078-9edf6c97d98d","Type":"ContainerStarted","Data":"24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef"} Jul 24 23:19:03.838189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.838059 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" event={"ID":"f53575bf-3458-4884-9078-9edf6c97d98d","Type":"ContainerStarted","Data":"9b1b38b4abcfadafa8d9bf6fc6105dbec183ba7389297f417a771cea8842af71"} Jul 24 23:19:03.858107 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:03.858012 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podStartSLOduration=0.857995359 podStartE2EDuration="857.995359ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:03.856050785 +0000 UTC m=+1603.131019906" watchObservedRunningTime="2026-07-24 23:19:03.857995359 +0000 UTC m=+1603.132964454" Jul 24 23:19:04.603795 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:04.603753 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:04.605210 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:04.605183 2562 patch_prober.go:28] interesting pod/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 24 23:19:04.605339 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:04.605232 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 24 23:19:05.607078 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:05.607044 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:05.846709 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:05.846661 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:05.847723 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:05.847704 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:06.014597 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.014512 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw"] Jul 24 23:19:06.017760 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.017740 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.020284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.020266 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a599b375\"" Jul 24 23:19:06.020351 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.020311 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a599b375-dockercfg-k6kqw\"" Jul 24 23:19:06.027955 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.027934 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw"] Jul 24 23:19:06.042271 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.042245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a599b375-57df96c7d-85lcw\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.042542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.042360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2r2x\" (UniqueName: \"kubernetes.io/projected/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-kube-api-access-l2r2x\") pod \"payload-processing-e2e-ait-a599b375-57df96c7d-85lcw\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.051176 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.051150 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v"] Jul 24 23:19:06.055161 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.055141 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.065372 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.065339 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v"] Jul 24 23:19:06.143374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.143332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqg6x\" (UniqueName: \"kubernetes.io/projected/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-kube-api-access-fqg6x\") pod \"payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.143374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.143374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2r2x\" (UniqueName: \"kubernetes.io/projected/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-kube-api-access-l2r2x\") pod \"payload-processing-e2e-ait-a599b375-57df96c7d-85lcw\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.143608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.143398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.143608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.143455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a599b375-57df96c7d-85lcw\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.144154 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.144120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a599b375-57df96c7d-85lcw\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.152115 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.152079 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2r2x\" (UniqueName: \"kubernetes.io/projected/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-kube-api-access-l2r2x\") pod \"payload-processing-e2e-ait-a599b375-57df96c7d-85lcw\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.244042 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.244002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.244238 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.244097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqg6x\" (UniqueName: \"kubernetes.io/projected/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-kube-api-access-fqg6x\") pod \"payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.244793 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.244769 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.252022 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.251997 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqg6x\" (UniqueName: \"kubernetes.io/projected/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-kube-api-access-fqg6x\") pod \"payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.330568 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.330533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.365971 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.365934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.468833 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.468803 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw"] Jul 24 23:19:06.471967 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:19:06.471941 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ca9e8c_4efe_4dd8_9bee_02622f4924e5.slice/crio-7dccd7243daf836130be52ffb88fa6b2ba02f3c76cab0e0bbb45918d69ef3ed2 WatchSource:0}: Error finding container 7dccd7243daf836130be52ffb88fa6b2ba02f3c76cab0e0bbb45918d69ef3ed2: Status 404 returned error can't find the container with id 7dccd7243daf836130be52ffb88fa6b2ba02f3c76cab0e0bbb45918d69ef3ed2 Jul 24 23:19:06.513228 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.513199 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v"] Jul 24 23:19:06.519703 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:19:06.519664 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7118ce5_29ba_4bb6_a4d1_6fd3d129cafa.slice/crio-61a27d826b21d0dd3b556a9961d54a48b7f70c1697e40f43e9a27724666bf5c8 WatchSource:0}: Error finding container 61a27d826b21d0dd3b556a9961d54a48b7f70c1697e40f43e9a27724666bf5c8: Status 404 returned error can't find the container with id 61a27d826b21d0dd3b556a9961d54a48b7f70c1697e40f43e9a27724666bf5c8 Jul 24 23:19:06.852231 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.852198 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" event={"ID":"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa","Type":"ContainerStarted","Data":"baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9"} Jul 24 23:19:06.852703 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.852260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" event={"ID":"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa","Type":"ContainerStarted","Data":"61a27d826b21d0dd3b556a9961d54a48b7f70c1697e40f43e9a27724666bf5c8"} Jul 24 23:19:06.852703 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.852287 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:06.853782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.853758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" event={"ID":"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5","Type":"ContainerStarted","Data":"50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada"} Jul 24 23:19:06.853909 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.853803 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" event={"ID":"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5","Type":"ContainerStarted","Data":"7dccd7243daf836130be52ffb88fa6b2ba02f3c76cab0e0bbb45918d69ef3ed2"} Jul 24 23:19:06.853909 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.853896 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:06.869015 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.868969 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" podStartSLOduration=0.868956733 podStartE2EDuration="868.956733ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:06.86688048 +0000 UTC m=+1606.141849577" watchObservedRunningTime="2026-07-24 23:19:06.868956733 +0000 UTC m=+1606.143925847" Jul 24 23:19:06.882961 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:06.882919 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" podStartSLOduration=1.8829091180000002 podStartE2EDuration="1.882909118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:06.882096601 +0000 UTC m=+1606.157065696" watchObservedRunningTime="2026-07-24 23:19:06.882909118 +0000 UTC m=+1606.157878212" Jul 24 23:19:17.859716 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:17.859667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:17.860176 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:17.859825 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:20.848258 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.848189 2562 patch_prober.go:28] interesting pod/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 24 23:19:20.848609 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.848251 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 24 23:19:20.914468 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.914435 2562 generic.go:356] "Generic (PLEG): container finished" podID="f53575bf-3458-4884-9078-9edf6c97d98d" containerID="24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef" exitCode=0 Jul 24 23:19:20.914610 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.914496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" event={"ID":"f53575bf-3458-4884-9078-9edf6c97d98d","Type":"ContainerDied","Data":"24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef"} Jul 24 23:19:20.914865 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.914849 2562 scope.go:117] "RemoveContainer" containerID="24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef" Jul 24 23:19:20.915814 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.915783 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:20.915886 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.915868 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:20.915922 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:20.915914 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:21.919925 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:21.919890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" event={"ID":"f53575bf-3458-4884-9078-9edf6c97d98d","Type":"ContainerStarted","Data":"da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b"} Jul 24 23:19:22.603865 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:22.603813 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:22.604112 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:22.604087 2562 patch_prober.go:28] interesting pod/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 24 23:19:22.604176 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:22.604140 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 24 23:19:23.603501 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:23.603465 2562 patch_prober.go:28] interesting pod/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 24 23:19:23.603928 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:23.603517 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 24 23:19:24.603226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:24.603191 2562 patch_prober.go:28] interesting pod/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 24 23:19:24.603430 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:24.603245 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 24 23:19:25.603352 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:25.603313 2562 patch_prober.go:28] interesting pod/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 24 23:19:25.603767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:25.603385 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 24 23:19:26.607283 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:26.607251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:26.607738 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:26.607598 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:26.608150 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:26.608133 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:41.209072 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.209035 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw"] Jul 24 23:19:41.209508 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.209314 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" podUID="f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" containerName="payload-processing" containerID="cri-o://50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada" gracePeriod=30 Jul 24 23:19:41.214989 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.214963 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v"] Jul 24 23:19:41.215346 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.215315 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" podUID="c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" containerName="payload-pre-processing" containerID="cri-o://baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9" gracePeriod=30 Jul 24 23:19:41.230735 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:19:41.229995 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ca9e8c_4efe_4dd8_9bee_02622f4924e5.slice/crio-50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada.scope\": RecentStats: unable to find data in memory cache]" Jul 24 23:19:41.484757 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.484737 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:41.487496 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.487478 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:41.554135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.554108 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-plugins-config-volume\") pod \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " Jul 24 23:19:41.554316 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.554203 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-plugins-config-volume\") pod \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " Jul 24 23:19:41.554316 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.554257 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqg6x\" (UniqueName: \"kubernetes.io/projected/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-kube-api-access-fqg6x\") pod \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\" (UID: \"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa\") " Jul 24 23:19:41.554316 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.554291 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2r2x\" (UniqueName: \"kubernetes.io/projected/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-kube-api-access-l2r2x\") pod \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\" (UID: \"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5\") " Jul 24 23:19:41.554529 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.554500 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" (UID: "c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:19:41.554591 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.554561 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" (UID: "f0ca9e8c-4efe-4dd8-9bee-02622f4924e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:19:41.556393 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.556364 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-kube-api-access-fqg6x" (OuterVolumeSpecName: "kube-api-access-fqg6x") pod "c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" (UID: "c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa"). InnerVolumeSpecName "kube-api-access-fqg6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:19:41.556484 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.556397 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-kube-api-access-l2r2x" (OuterVolumeSpecName: "kube-api-access-l2r2x") pod "f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" (UID: "f0ca9e8c-4efe-4dd8-9bee-02622f4924e5"). InnerVolumeSpecName "kube-api-access-l2r2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:19:41.654894 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.654869 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqg6x\" (UniqueName: \"kubernetes.io/projected/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-kube-api-access-fqg6x\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:41.654894 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.654889 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2r2x\" (UniqueName: \"kubernetes.io/projected/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-kube-api-access-l2r2x\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:41.654894 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.654899 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:41.655061 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:41.654908 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:42.016573 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.016538 2562 generic.go:356] "Generic (PLEG): container finished" podID="c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" containerID="baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9" exitCode=0 Jul 24 23:19:42.016780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.016596 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" Jul 24 23:19:42.016780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.016635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" event={"ID":"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa","Type":"ContainerDied","Data":"baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9"} Jul 24 23:19:42.016780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.016731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v" event={"ID":"c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa","Type":"ContainerDied","Data":"61a27d826b21d0dd3b556a9961d54a48b7f70c1697e40f43e9a27724666bf5c8"} Jul 24 23:19:42.016780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.016759 2562 scope.go:117] "RemoveContainer" containerID="baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9" Jul 24 23:19:42.017887 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.017868 2562 generic.go:356] "Generic (PLEG): container finished" podID="f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" containerID="50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada" exitCode=0 Jul 24 23:19:42.017955 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.017894 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" event={"ID":"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5","Type":"ContainerDied","Data":"50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada"} Jul 24 23:19:42.017955 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.017937 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" event={"ID":"f0ca9e8c-4efe-4dd8-9bee-02622f4924e5","Type":"ContainerDied","Data":"7dccd7243daf836130be52ffb88fa6b2ba02f3c76cab0e0bbb45918d69ef3ed2"} Jul 24 23:19:42.017955 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.017941 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw" Jul 24 23:19:42.025035 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.025015 2562 scope.go:117] "RemoveContainer" containerID="baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9" Jul 24 23:19:42.025264 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:19:42.025244 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9\": container with ID starting with baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9 not found: ID does not exist" containerID="baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9" Jul 24 23:19:42.025317 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.025270 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9"} err="failed to get container status \"baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9\": rpc error: code = NotFound desc = could not find container \"baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9\": container with ID starting with baaaf8256ade43cab5ddaca3f657c5271f1cea172a4d2aaa5b3f3da67a8d9ac9 not found: ID does not exist" Jul 24 23:19:42.025317 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.025286 2562 scope.go:117] "RemoveContainer" containerID="50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada" Jul 24 23:19:42.033020 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.032997 2562 scope.go:117] "RemoveContainer" containerID="50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada" Jul 24 23:19:42.033251 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:19:42.033232 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada\": container with ID starting with 50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada not found: ID does not exist" containerID="50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada" Jul 24 23:19:42.033301 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.033260 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada"} err="failed to get container status \"50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada\": rpc error: code = NotFound desc = could not find container \"50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada\": container with ID starting with 50b77a6f5583d1207fa022f0823dea69918a1359f24dfa402237a8bcda313ada not found: ID does not exist" Jul 24 23:19:42.041968 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.041946 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v"] Jul 24 23:19:42.046442 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.046419 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a599b375-6b8fc9d964-4g68v"] Jul 24 23:19:42.061042 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.060988 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw"] Jul 24 23:19:42.062936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:42.062917 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a599b375-57df96c7d-85lcw"] Jul 24 23:19:43.299408 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:43.299378 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" path="/var/lib/kubelet/pods/c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa/volumes" Jul 24 23:19:43.299819 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:43.299737 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" path="/var/lib/kubelet/pods/f0ca9e8c-4efe-4dd8-9bee-02622f4924e5/volumes" Jul 24 23:19:45.981071 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:45.981037 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr"] Jul 24 23:19:45.981542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:45.981241 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" containerID="cri-o://da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b" gracePeriod=30 Jul 24 23:19:46.979907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.979877 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv"] Jul 24 23:19:46.980234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.980217 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" containerName="payload-pre-processing" Jul 24 23:19:46.980292 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.980238 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" containerName="payload-pre-processing" Jul 24 23:19:46.980292 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.980258 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" containerName="payload-processing" Jul 24 23:19:46.980292 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.980265 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" containerName="payload-processing" Jul 24 23:19:46.980391 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.980330 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7118ce5-29ba-4bb6-a4d1-6fd3d129cafa" containerName="payload-pre-processing" Jul 24 23:19:46.980391 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.980341 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ca9e8c-4efe-4dd8-9bee-02622f4924e5" containerName="payload-processing" Jul 24 23:19:46.985007 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.984986 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:46.988630 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.988610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-629f57d4-gw-openshift-default-dockercfg-ddvd8\"" Jul 24 23:19:46.998782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:46.998621 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv"] Jul 24 23:19:47.000333 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-certs\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000430 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000348 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-data\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000430 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-token\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000546 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1aa34-9217-41ff-b485-28cd90d55219-istiod-ca-cert\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000546 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxl74\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-kube-api-access-kxl74\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000546 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-socket\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000707 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-credential-socket\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000707 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-envoy\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.000707 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.000689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-podinfo\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.101534 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxl74\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-kube-api-access-kxl74\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.101737 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-socket\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.101737 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-credential-socket\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.101737 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-envoy\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.101737 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-podinfo\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.101737 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-certs\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102023 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-data\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102023 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.101949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-token\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102023 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.102019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1aa34-9217-41ff-b485-28cd90d55219-istiod-ca-cert\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102173 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.102086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-credential-socket\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102173 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.102129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-socket\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102173 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.102148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-data\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102442 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.102423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-certs\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.102789 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.102768 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1aa34-9217-41ff-b485-28cd90d55219-istiod-ca-cert\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.103992 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.103971 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-envoy\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.104265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.104244 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-podinfo\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.110487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.110464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxl74\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-kube-api-access-kxl74\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.110855 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.110835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-token\") pod \"e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.294866 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.294775 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:47.429544 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.427334 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv"] Jul 24 23:19:47.430626 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:19:47.430591 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc1aa34_9217_41ff_b485_28cd90d55219.slice/crio-c0770cc36afef4797aef352f4f32f716d9bccd03e63471a10bed1addc1a9a878 WatchSource:0}: Error finding container c0770cc36afef4797aef352f4f32f716d9bccd03e63471a10bed1addc1a9a878: Status 404 returned error can't find the container with id c0770cc36afef4797aef352f4f32f716d9bccd03e63471a10bed1addc1a9a878 Jul 24 23:19:47.432906 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.432876 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:47.432978 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.432939 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:47.432978 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:47.432968 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:19:48.048109 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:48.048069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" event={"ID":"cdc1aa34-9217-41ff-b485-28cd90d55219","Type":"ContainerStarted","Data":"2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38"} Jul 24 23:19:48.048493 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:48.048131 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" event={"ID":"cdc1aa34-9217-41ff-b485-28cd90d55219","Type":"ContainerStarted","Data":"c0770cc36afef4797aef352f4f32f716d9bccd03e63471a10bed1addc1a9a878"} Jul 24 23:19:48.069870 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:48.069827 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" podStartSLOduration=2.069811815 podStartE2EDuration="2.069811815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:48.067229781 +0000 UTC m=+1647.342198877" watchObservedRunningTime="2026-07-24 23:19:48.069811815 +0000 UTC m=+1647.344780970" Jul 24 23:19:48.295728 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:48.295690 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:48.300465 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:48.300402 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:49.052408 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:49.052378 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:49.053376 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:49.053358 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:19:50.397719 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.397685 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn"] Jul 24 23:19:50.400895 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.400874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.403477 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.403455 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-629f57d4-dockercfg-c4tqw\"" Jul 24 23:19:50.403589 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.403489 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-629f57d4\"" Jul 24 23:19:50.411078 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.411044 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn"] Jul 24 23:19:50.432494 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.432468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0849ce08-f239-4ee0-a664-1f8e805b4dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.432603 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.432578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqlw\" (UniqueName: \"kubernetes.io/projected/0849ce08-f239-4ee0-a664-1f8e805b4dd5-kube-api-access-plqlw\") pod \"payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.533618 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.533583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqlw\" (UniqueName: \"kubernetes.io/projected/0849ce08-f239-4ee0-a664-1f8e805b4dd5-kube-api-access-plqlw\") pod \"payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.533618 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.533623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0849ce08-f239-4ee0-a664-1f8e805b4dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.534299 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.534280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0849ce08-f239-4ee0-a664-1f8e805b4dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.541490 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.541467 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqlw\" (UniqueName: \"kubernetes.io/projected/0849ce08-f239-4ee0-a664-1f8e805b4dd5-kube-api-access-plqlw\") pod \"payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.712529 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.712453 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:50.840776 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:50.840750 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn"] Jul 24 23:19:50.842696 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:19:50.842656 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0849ce08_f239_4ee0_a664_1f8e805b4dd5.slice/crio-f674e33084e95026da703b1dfb9cd4ffb322aea3cd1fbba04dcdf25663f8f44d WatchSource:0}: Error finding container f674e33084e95026da703b1dfb9cd4ffb322aea3cd1fbba04dcdf25663f8f44d: Status 404 returned error can't find the container with id f674e33084e95026da703b1dfb9cd4ffb322aea3cd1fbba04dcdf25663f8f44d Jul 24 23:19:51.063243 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.063207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" event={"ID":"0849ce08-f239-4ee0-a664-1f8e805b4dd5","Type":"ContainerStarted","Data":"ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c"} Jul 24 23:19:51.063386 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.063268 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" event={"ID":"0849ce08-f239-4ee0-a664-1f8e805b4dd5","Type":"ContainerStarted","Data":"f674e33084e95026da703b1dfb9cd4ffb322aea3cd1fbba04dcdf25663f8f44d"} Jul 24 23:19:51.079321 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.079272 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" podStartSLOduration=1.079257376 podStartE2EDuration="1.079257376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:51.078296928 +0000 UTC m=+1650.353266051" watchObservedRunningTime="2026-07-24 23:19:51.079257376 +0000 UTC m=+1650.354226470" Jul 24 23:19:51.226287 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.226264 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:51.342136 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-data\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342136 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342136 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-certs\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342171 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-envoy\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342202 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-socket\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342232 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mll6x\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-kube-api-access-mll6x\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342284 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-istio-token\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53575bf-3458-4884-9078-9edf6c97d98d-istiod-ca-cert\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342596 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342366 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-credential-socket\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342596 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342399 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53575bf-3458-4884-9078-9edf6c97d98d-istio-podinfo\") pod \"f53575bf-3458-4884-9078-9edf6c97d98d\" (UID: \"f53575bf-3458-4884-9078-9edf6c97d98d\") " Jul 24 23:19:51.342596 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342401 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:19:51.342761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342566 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:19:51.342761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342712 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.342761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342731 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.342761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342732 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53575bf-3458-4884-9078-9edf6c97d98d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:19:51.342956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342811 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:19:51.343008 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.342962 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:19:51.344473 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.344450 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:19:51.344577 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.344536 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f53575bf-3458-4884-9078-9edf6c97d98d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:19:51.344814 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.344785 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:19:51.344894 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.344838 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-kube-api-access-mll6x" (OuterVolumeSpecName: "kube-api-access-mll6x") pod "f53575bf-3458-4884-9078-9edf6c97d98d" (UID: "f53575bf-3458-4884-9078-9edf6c97d98d"). InnerVolumeSpecName "kube-api-access-mll6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:19:51.444054 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444022 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.444054 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444046 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53575bf-3458-4884-9078-9edf6c97d98d-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.444054 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444058 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.444458 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444071 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53575bf-3458-4884-9078-9edf6c97d98d-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.444458 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444084 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.444458 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444092 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53575bf-3458-4884-9078-9edf6c97d98d-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:51.444458 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:51.444100 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mll6x\" (UniqueName: \"kubernetes.io/projected/f53575bf-3458-4884-9078-9edf6c97d98d-kube-api-access-mll6x\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:19:52.070447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.070399 2562 generic.go:356] "Generic (PLEG): container finished" podID="f53575bf-3458-4884-9078-9edf6c97d98d" containerID="da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b" exitCode=0 Jul 24 23:19:52.070447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.070446 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" Jul 24 23:19:52.070653 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.070485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" event={"ID":"f53575bf-3458-4884-9078-9edf6c97d98d","Type":"ContainerDied","Data":"da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b"} Jul 24 23:19:52.070653 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.070547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr" event={"ID":"f53575bf-3458-4884-9078-9edf6c97d98d","Type":"ContainerDied","Data":"9b1b38b4abcfadafa8d9bf6fc6105dbec183ba7389297f417a771cea8842af71"} Jul 24 23:19:52.070653 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.070574 2562 scope.go:117] "RemoveContainer" containerID="da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b" Jul 24 23:19:52.070904 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.070884 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:19:52.079495 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.079471 2562 scope.go:117] "RemoveContainer" containerID="24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef" Jul 24 23:19:52.086895 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.086880 2562 scope.go:117] "RemoveContainer" containerID="da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b" Jul 24 23:19:52.087132 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:19:52.087116 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b\": container with ID starting with da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b not found: ID does not exist" containerID="da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b" Jul 24 23:19:52.087180 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.087138 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b"} err="failed to get container status \"da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b\": rpc error: code = NotFound desc = could not find container \"da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b\": container with ID starting with da831793244a9d7fd46e4734bdd55802dce43df010657202310c3aeea2808d5b not found: ID does not exist" Jul 24 23:19:52.087180 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.087154 2562 scope.go:117] "RemoveContainer" containerID="24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef" Jul 24 23:19:52.087358 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:19:52.087339 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef\": container with ID starting with 24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef not found: ID does not exist" containerID="24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef" Jul 24 23:19:52.087391 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.087362 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef"} err="failed to get container status \"24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef\": rpc error: code = NotFound desc = could not find container \"24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef\": container with ID starting with 24814c70e95a1a8d6c1ca633897703a5c8c486e89c52318346b38929a933f2ef not found: ID does not exist" Jul 24 23:19:52.092338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.092317 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr"] Jul 24 23:19:52.097710 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:52.097664 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a599b375-openshift-default-69f65cd45d-pfxnr"] Jul 24 23:19:53.299239 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:19:53.299199 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" path="/var/lib/kubelet/pods/f53575bf-3458-4884-9078-9edf6c97d98d/volumes" Jul 24 23:20:03.078860 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.078831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:20:03.122629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.122597 2562 generic.go:356] "Generic (PLEG): container finished" podID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerID="2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38" exitCode=0 Jul 24 23:20:03.122782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.122688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" event={"ID":"cdc1aa34-9217-41ff-b485-28cd90d55219","Type":"ContainerDied","Data":"2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38"} Jul 24 23:20:03.123056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.123041 2562 scope.go:117] "RemoveContainer" containerID="2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38" Jul 24 23:20:03.124076 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.124051 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:03.124136 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.124114 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:03.124177 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:03.124145 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:04.053041 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:04.053000 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:04.131917 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:04.131873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" event={"ID":"cdc1aa34-9217-41ff-b485-28cd90d55219","Type":"ContainerStarted","Data":"7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845"} Jul 24 23:20:04.295661 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:04.295631 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:04.296802 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:04.296776 2562 patch_prober.go:28] interesting pod/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 24 23:20:04.296924 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:04.296841 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 24 23:20:05.295951 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:05.295917 2562 patch_prober.go:28] interesting pod/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 24 23:20:05.296325 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:05.295968 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 24 23:20:06.295531 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:06.295495 2562 patch_prober.go:28] interesting pod/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 24 23:20:06.295733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:06.295543 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 24 23:20:07.295422 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:07.295382 2562 patch_prober.go:28] interesting pod/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Jul 24 23:20:07.295830 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:07.295444 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Jul 24 23:20:08.299124 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:08.299092 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:08.299541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:08.299441 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:08.300134 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:08.300114 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:14.539719 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.539666 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d"] Jul 24 23:20:14.540194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.539988 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" Jul 24 23:20:14.540194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.539999 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" Jul 24 23:20:14.540194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.540007 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" Jul 24 23:20:14.540194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.540013 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" Jul 24 23:20:14.540194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.540061 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" Jul 24 23:20:14.540194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.540069 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53575bf-3458-4884-9078-9edf6c97d98d" containerName="istio-proxy" Jul 24 23:20:14.542916 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.542894 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:14.545767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.545743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h9zqg\"" Jul 24 23:20:14.545889 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.545826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:20:14.551005 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.550599 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d"] Jul 24 23:20:14.639818 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.639775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f68984-a39b-455f-a102-61e28e4ee729-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d\" (UID: \"30f68984-a39b-455f-a102-61e28e4ee729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:14.740830 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.740799 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f68984-a39b-455f-a102-61e28e4ee729-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d\" (UID: \"30f68984-a39b-455f-a102-61e28e4ee729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:14.741389 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.741369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f68984-a39b-455f-a102-61e28e4ee729-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d\" (UID: \"30f68984-a39b-455f-a102-61e28e4ee729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:14.854281 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.854255 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:14.983743 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:14.983695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d"] Jul 24 23:20:14.986525 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:20:14.986498 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f68984_a39b_455f_a102_61e28e4ee729.slice/crio-c5586a392eea8974e303a21210e871c16f8ff5f95783f1bc75bd4b9a68b53596 WatchSource:0}: Error finding container c5586a392eea8974e303a21210e871c16f8ff5f95783f1bc75bd4b9a68b53596: Status 404 returned error can't find the container with id c5586a392eea8974e303a21210e871c16f8ff5f95783f1bc75bd4b9a68b53596 Jul 24 23:20:15.187654 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:15.187617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" event={"ID":"30f68984-a39b-455f-a102-61e28e4ee729","Type":"ContainerStarted","Data":"c5586a392eea8974e303a21210e871c16f8ff5f95783f1bc75bd4b9a68b53596"} Jul 24 23:20:16.194387 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:16.194349 2562 generic.go:356] "Generic (PLEG): container finished" podID="30f68984-a39b-455f-a102-61e28e4ee729" containerID="dd43a10cacd22021d61225cca8853eb16fe1086a516eb66fd23ae92f69ab6cb9" exitCode=0 Jul 24 23:20:16.194826 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:16.194436 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" event={"ID":"30f68984-a39b-455f-a102-61e28e4ee729","Type":"ContainerDied","Data":"dd43a10cacd22021d61225cca8853eb16fe1086a516eb66fd23ae92f69ab6cb9"} Jul 24 23:20:17.323301 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:17.323281 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:17.359948 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:17.359924 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f68984-a39b-455f-a102-61e28e4ee729-maas-api-service-ca\") pod \"30f68984-a39b-455f-a102-61e28e4ee729\" (UID: \"30f68984-a39b-455f-a102-61e28e4ee729\") " Jul 24 23:20:17.360252 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:17.360222 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f68984-a39b-455f-a102-61e28e4ee729-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30f68984-a39b-455f-a102-61e28e4ee729" (UID: "30f68984-a39b-455f-a102-61e28e4ee729"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:20:17.461553 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:17.461479 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f68984-a39b-455f-a102-61e28e4ee729-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:18.209718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:18.209670 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" Jul 24 23:20:18.209718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:18.209696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d" event={"ID":"30f68984-a39b-455f-a102-61e28e4ee729","Type":"ContainerDied","Data":"c5586a392eea8974e303a21210e871c16f8ff5f95783f1bc75bd4b9a68b53596"} Jul 24 23:20:18.209984 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:18.209765 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5586a392eea8974e303a21210e871c16f8ff5f95783f1bc75bd4b9a68b53596" Jul 24 23:20:24.983725 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:24.983665 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn"] Jul 24 23:20:24.984205 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:24.983933 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" podUID="0849ce08-f239-4ee0-a664-1f8e805b4dd5" containerName="payload-pre-processing" containerID="cri-o://ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c" gracePeriod=30 Jul 24 23:20:25.233906 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.233842 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:20:25.240595 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.240560 2562 generic.go:356] "Generic (PLEG): container finished" podID="0849ce08-f239-4ee0-a664-1f8e805b4dd5" containerID="ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c" exitCode=0 Jul 24 23:20:25.240761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.240603 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" Jul 24 23:20:25.240761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.240607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" event={"ID":"0849ce08-f239-4ee0-a664-1f8e805b4dd5","Type":"ContainerDied","Data":"ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c"} Jul 24 23:20:25.240761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.240669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn" event={"ID":"0849ce08-f239-4ee0-a664-1f8e805b4dd5","Type":"ContainerDied","Data":"f674e33084e95026da703b1dfb9cd4ffb322aea3cd1fbba04dcdf25663f8f44d"} Jul 24 23:20:25.240761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.240732 2562 scope.go:117] "RemoveContainer" containerID="ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c" Jul 24 23:20:25.248909 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.248890 2562 scope.go:117] "RemoveContainer" containerID="ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c" Jul 24 23:20:25.249245 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:20:25.249217 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c\": container with ID starting with ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c not found: ID does not exist" containerID="ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c" Jul 24 23:20:25.249353 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.249255 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c"} err="failed to get container status \"ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c\": rpc error: code = NotFound desc = could not find container \"ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c\": container with ID starting with ebdcecf083cde22b1f50c19fa077053ea63dcbe929203803b6cdbd305a0c3e7c not found: ID does not exist" Jul 24 23:20:25.327873 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.327844 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0849ce08-f239-4ee0-a664-1f8e805b4dd5-plugins-config-volume\") pod \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " Jul 24 23:20:25.328040 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.327887 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plqlw\" (UniqueName: \"kubernetes.io/projected/0849ce08-f239-4ee0-a664-1f8e805b4dd5-kube-api-access-plqlw\") pod \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\" (UID: \"0849ce08-f239-4ee0-a664-1f8e805b4dd5\") " Jul 24 23:20:25.328232 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.328208 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0849ce08-f239-4ee0-a664-1f8e805b4dd5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0849ce08-f239-4ee0-a664-1f8e805b4dd5" (UID: "0849ce08-f239-4ee0-a664-1f8e805b4dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:20:25.329919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.329887 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0849ce08-f239-4ee0-a664-1f8e805b4dd5-kube-api-access-plqlw" (OuterVolumeSpecName: "kube-api-access-plqlw") pod "0849ce08-f239-4ee0-a664-1f8e805b4dd5" (UID: "0849ce08-f239-4ee0-a664-1f8e805b4dd5"). InnerVolumeSpecName "kube-api-access-plqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:20:25.429213 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.429183 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0849ce08-f239-4ee0-a664-1f8e805b4dd5-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:25.429213 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.429206 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plqlw\" (UniqueName: \"kubernetes.io/projected/0849ce08-f239-4ee0-a664-1f8e805b4dd5-kube-api-access-plqlw\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:25.563301 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.563273 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn"] Jul 24 23:20:25.563551 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:20:25.563532 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0849ce08_f239_4ee0_a664_1f8e805b4dd5.slice/crio-f674e33084e95026da703b1dfb9cd4ffb322aea3cd1fbba04dcdf25663f8f44d\": RecentStats: unable to find data in memory cache]" Jul 24 23:20:25.566784 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:25.566761 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-629f57d4-7b8bbcf8d4-mghxn"] Jul 24 23:20:27.299406 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:27.299374 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0849ce08-f239-4ee0-a664-1f8e805b4dd5" path="/var/lib/kubelet/pods/0849ce08-f239-4ee0-a664-1f8e805b4dd5/volumes" Jul 24 23:20:29.618201 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:29.618166 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d"] Jul 24 23:20:29.621252 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:29.621229 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-629f57d4-gbd8d"] Jul 24 23:20:29.744382 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:29.744348 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv"] Jul 24 23:20:29.744567 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:29.744544 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" containerID="cri-o://7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845" gracePeriod=30 Jul 24 23:20:30.550961 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.550931 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85"] Jul 24 23:20:30.551270 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.551258 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f68984-a39b-455f-a102-61e28e4ee729" containerName="revoke-keys" Jul 24 23:20:30.551313 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.551273 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f68984-a39b-455f-a102-61e28e4ee729" containerName="revoke-keys" Jul 24 23:20:30.551313 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.551285 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0849ce08-f239-4ee0-a664-1f8e805b4dd5" containerName="payload-pre-processing" Jul 24 23:20:30.551313 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.551291 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0849ce08-f239-4ee0-a664-1f8e805b4dd5" containerName="payload-pre-processing" Jul 24 23:20:30.551403 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.551334 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0849ce08-f239-4ee0-a664-1f8e805b4dd5" containerName="payload-pre-processing" Jul 24 23:20:30.551403 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.551344 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f68984-a39b-455f-a102-61e28e4ee729" containerName="revoke-keys" Jul 24 23:20:30.555782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.555755 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.558465 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.558443 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2af5eeed-openshift-default-dockercfg-hxnlq\"" Jul 24 23:20:30.567578 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.567553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85"] Jul 24 23:20:30.673896 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.673862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-socket\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.674462 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.674439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-data\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.674608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.674594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-token\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.674757 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.674741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istiod-ca-cert\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.674901 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.674888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-certs\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.675010 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.674998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-podinfo\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.675121 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.675109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-envoy\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.675238 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.675226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-credential-socket\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.675332 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.675318 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftx7h\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-kube-api-access-ftx7h\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.776779 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.776750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-token\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.776779 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.776783 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istiod-ca-cert\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777010 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.776810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-certs\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777010 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.776944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-podinfo\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777010 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-envoy\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777172 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-credential-socket\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777172 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftx7h\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-kube-api-access-ftx7h\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777172 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777125 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-certs\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777172 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-socket\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777365 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-data\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777423 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-credential-socket\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777533 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-data\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777590 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istiod-ca-cert\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.777625 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.777587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-socket\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.779213 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.779191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-envoy\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.779409 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.779392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-podinfo\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.788931 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.788906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftx7h\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-kube-api-access-ftx7h\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.788931 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.788919 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-token\") pod \"e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.867662 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.867638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:30.999862 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:30.999838 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85"] Jul 24 23:20:31.001717 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:20:31.001667 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cbec88_dbd5_4e43_92a4_ee8bd594540c.slice/crio-653ddb76019992ce49259813257b07998d6bb550f598b34107da4679d46fc505 WatchSource:0}: Error finding container 653ddb76019992ce49259813257b07998d6bb550f598b34107da4679d46fc505: Status 404 returned error can't find the container with id 653ddb76019992ce49259813257b07998d6bb550f598b34107da4679d46fc505 Jul 24 23:20:31.003780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.003752 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:31.003849 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.003815 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:31.003849 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.003841 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:31.267849 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.267765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" event={"ID":"01cbec88-dbd5-4e43-92a4-ee8bd594540c","Type":"ContainerStarted","Data":"7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909"} Jul 24 23:20:31.267849 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.267828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" event={"ID":"01cbec88-dbd5-4e43-92a4-ee8bd594540c","Type":"ContainerStarted","Data":"653ddb76019992ce49259813257b07998d6bb550f598b34107da4679d46fc505"} Jul 24 23:20:31.287946 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.287886 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" podStartSLOduration=1.2878654379999999 podStartE2EDuration="1.287865438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:20:31.286951427 +0000 UTC m=+1690.561920532" watchObservedRunningTime="2026-07-24 23:20:31.287865438 +0000 UTC m=+1690.562834534" Jul 24 23:20:31.299101 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.299073 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f68984-a39b-455f-a102-61e28e4ee729" path="/var/lib/kubelet/pods/30f68984-a39b-455f-a102-61e28e4ee729/volumes" Jul 24 23:20:31.868392 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:31.868360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:32.873180 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:32.873153 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:33.277002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:33.276907 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:33.277974 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:33.277954 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:35.001353 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.001330 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:35.115328 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115304 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-envoy\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115341 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-certs\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115381 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-token\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115399 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxl74\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-kube-api-access-kxl74\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115650 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115512 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-podinfo\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115650 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115549 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-data\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115650 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115573 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-credential-socket\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115650 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115607 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1aa34-9217-41ff-b485-28cd90d55219-istiod-ca-cert\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115867 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115690 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-socket\") pod \"cdc1aa34-9217-41ff-b485-28cd90d55219\" (UID: \"cdc1aa34-9217-41ff-b485-28cd90d55219\") " Jul 24 23:20:35.115867 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115698 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:20:35.115980 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.115942 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:20:35.116065 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.116034 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.116065 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.116059 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.116180 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.116069 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc1aa34-9217-41ff-b485-28cd90d55219-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:20:35.116647 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.116146 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:20:35.117023 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.116999 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:20:35.117615 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.117592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-kube-api-access-kxl74" (OuterVolumeSpecName: "kube-api-access-kxl74") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "kube-api-access-kxl74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:20:35.117805 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.117777 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:20:35.117956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.117938 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:20:35.118049 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.118031 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdc1aa34-9217-41ff-b485-28cd90d55219" (UID: "cdc1aa34-9217-41ff-b485-28cd90d55219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:20:35.216892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216866 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.216892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216888 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.216892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216897 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.217048 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216907 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxl74\" (UniqueName: \"kubernetes.io/projected/cdc1aa34-9217-41ff-b485-28cd90d55219-kube-api-access-kxl74\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.217048 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216916 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.217048 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216924 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdc1aa34-9217-41ff-b485-28cd90d55219-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.217048 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.216932 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdc1aa34-9217-41ff-b485-28cd90d55219-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:20:35.286549 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.286513 2562 generic.go:356] "Generic (PLEG): container finished" podID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerID="7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845" exitCode=0 Jul 24 23:20:35.286699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.286568 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" Jul 24 23:20:35.286699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.286601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" event={"ID":"cdc1aa34-9217-41ff-b485-28cd90d55219","Type":"ContainerDied","Data":"7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845"} Jul 24 23:20:35.286699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.286656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv" event={"ID":"cdc1aa34-9217-41ff-b485-28cd90d55219","Type":"ContainerDied","Data":"c0770cc36afef4797aef352f4f32f716d9bccd03e63471a10bed1addc1a9a878"} Jul 24 23:20:35.286699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.286692 2562 scope.go:117] "RemoveContainer" containerID="7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845" Jul 24 23:20:35.295990 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.295974 2562 scope.go:117] "RemoveContainer" containerID="2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38" Jul 24 23:20:35.309610 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.309591 2562 scope.go:117] "RemoveContainer" containerID="7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845" Jul 24 23:20:35.309874 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:20:35.309852 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845\": container with ID starting with 7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845 not found: ID does not exist" containerID="7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845" Jul 24 23:20:35.309925 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.309886 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845"} err="failed to get container status \"7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845\": rpc error: code = NotFound desc = could not find container \"7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845\": container with ID starting with 7e17f4dc816ddf00655cde7c08c4858ffc8876bb47bdef962189b1f20b7c5845 not found: ID does not exist" Jul 24 23:20:35.309925 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.309912 2562 scope.go:117] "RemoveContainer" containerID="2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38" Jul 24 23:20:35.310229 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:20:35.310195 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38\": container with ID starting with 2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38 not found: ID does not exist" containerID="2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38" Jul 24 23:20:35.310330 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.310237 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38"} err="failed to get container status \"2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38\": rpc error: code = NotFound desc = could not find container \"2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38\": container with ID starting with 2d14195c0b118a4a00a07725791ed6960769d753ebf5602eb9c6538c5d0adf38 not found: ID does not exist" Jul 24 23:20:35.312406 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.312385 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv"] Jul 24 23:20:35.316091 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:35.316072 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-629f57d4-gw-openshift-default-767d67fbcb-g8zsv"] Jul 24 23:20:37.298741 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:37.298705 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" path="/var/lib/kubelet/pods/cdc1aa34-9217-41ff-b485-28cd90d55219/volumes" Jul 24 23:20:47.336685 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:47.336648 2562 generic.go:356] "Generic (PLEG): container finished" podID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerID="7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909" exitCode=0 Jul 24 23:20:47.337151 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:47.336720 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" event={"ID":"01cbec88-dbd5-4e43-92a4-ee8bd594540c","Type":"ContainerDied","Data":"7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909"} Jul 24 23:20:47.337151 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:47.337052 2562 scope.go:117] "RemoveContainer" containerID="7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909" Jul 24 23:20:47.337947 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:47.337922 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:47.338002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:47.337984 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:47.338037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:47.338015 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:20:48.277513 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:48.277460 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:48.345769 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:48.345727 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" event={"ID":"01cbec88-dbd5-4e43-92a4-ee8bd594540c","Type":"ContainerStarted","Data":"1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66"} Jul 24 23:20:48.868617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:48.868570 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:48.869745 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:48.869720 2562 patch_prober.go:28] interesting pod/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 24 23:20:48.869889 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:48.869777 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 24 23:20:49.868814 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:49.868774 2562 patch_prober.go:28] interesting pod/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 24 23:20:49.869189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:49.868832 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 24 23:20:50.868412 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:50.868376 2562 patch_prober.go:28] interesting pod/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 24 23:20:50.868585 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:50.868440 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 24 23:20:51.868748 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:51.868711 2562 patch_prober.go:28] interesting pod/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 24 23:20:51.869139 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:51.868767 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 24 23:20:52.871780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:52.871742 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:52.872247 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:52.872137 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:52.872709 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:52.872690 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:20:58.490820 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.490783 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q"] Jul 24 23:20:58.491187 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.491099 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" Jul 24 23:20:58.491187 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.491110 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" Jul 24 23:20:58.491187 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.491122 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" Jul 24 23:20:58.491187 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.491128 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" Jul 24 23:20:58.491187 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.491189 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" Jul 24 23:20:58.491347 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.491198 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc1aa34-9217-41ff-b485-28cd90d55219" containerName="istio-proxy" Jul 24 23:20:58.493924 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.493904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:20:58.496378 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.496356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:20:58.496501 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.496444 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h9zqg\"" Jul 24 23:20:58.500342 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.500319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q"] Jul 24 23:20:58.508764 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.508738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592f82-60c9-4800-a9a8-eb4e43acb7a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q\" (UID: \"12592f82-60c9-4800-a9a8-eb4e43acb7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:20:58.609251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.609216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592f82-60c9-4800-a9a8-eb4e43acb7a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q\" (UID: \"12592f82-60c9-4800-a9a8-eb4e43acb7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:20:58.609872 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.609854 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592f82-60c9-4800-a9a8-eb4e43acb7a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q\" (UID: \"12592f82-60c9-4800-a9a8-eb4e43acb7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:20:58.806138 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.806059 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:20:58.938241 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:58.938211 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q"] Jul 24 23:20:58.940600 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:20:58.940572 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12592f82_60c9_4800_a9a8_eb4e43acb7a2.slice/crio-07d147f657f659936553997d90ab95206f51fd20f8075894589a3c3448ac89dc WatchSource:0}: Error finding container 07d147f657f659936553997d90ab95206f51fd20f8075894589a3c3448ac89dc: Status 404 returned error can't find the container with id 07d147f657f659936553997d90ab95206f51fd20f8075894589a3c3448ac89dc Jul 24 23:20:59.392448 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:59.392408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" event={"ID":"12592f82-60c9-4800-a9a8-eb4e43acb7a2","Type":"ContainerStarted","Data":"adf2072c4ed08b4b6cdecc4db5e6e6dceb60b8c6c0a21e4f2ddaf22f6f0cf45c"} Jul 24 23:20:59.392580 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:59.392470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" event={"ID":"12592f82-60c9-4800-a9a8-eb4e43acb7a2","Type":"ContainerStarted","Data":"07d147f657f659936553997d90ab95206f51fd20f8075894589a3c3448ac89dc"} Jul 24 23:20:59.408538 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:20:59.408459 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" podStartSLOduration=1.4084450849999999 podStartE2EDuration="1.408445085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:20:59.406725753 +0000 UTC m=+1718.681694847" watchObservedRunningTime="2026-07-24 23:20:59.408445085 +0000 UTC m=+1718.683414182" Jul 24 23:21:00.397634 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:00.397597 2562 generic.go:356] "Generic (PLEG): container finished" podID="12592f82-60c9-4800-a9a8-eb4e43acb7a2" containerID="adf2072c4ed08b4b6cdecc4db5e6e6dceb60b8c6c0a21e4f2ddaf22f6f0cf45c" exitCode=0 Jul 24 23:21:00.398041 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:00.397696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" event={"ID":"12592f82-60c9-4800-a9a8-eb4e43acb7a2","Type":"ContainerDied","Data":"adf2072c4ed08b4b6cdecc4db5e6e6dceb60b8c6c0a21e4f2ddaf22f6f0cf45c"} Jul 24 23:21:01.531949 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:01.531923 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:21:01.632110 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:01.632079 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592f82-60c9-4800-a9a8-eb4e43acb7a2-maas-api-service-ca\") pod \"12592f82-60c9-4800-a9a8-eb4e43acb7a2\" (UID: \"12592f82-60c9-4800-a9a8-eb4e43acb7a2\") " Jul 24 23:21:01.632404 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:01.632383 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12592f82-60c9-4800-a9a8-eb4e43acb7a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12592f82-60c9-4800-a9a8-eb4e43acb7a2" (UID: "12592f82-60c9-4800-a9a8-eb4e43acb7a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:21:01.733079 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:01.732997 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592f82-60c9-4800-a9a8-eb4e43acb7a2-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:02.408401 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:02.408370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" event={"ID":"12592f82-60c9-4800-a9a8-eb4e43acb7a2","Type":"ContainerDied","Data":"07d147f657f659936553997d90ab95206f51fd20f8075894589a3c3448ac89dc"} Jul 24 23:21:02.408401 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:02.408401 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q" Jul 24 23:21:02.408599 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:02.408416 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d147f657f659936553997d90ab95206f51fd20f8075894589a3c3448ac89dc" Jul 24 23:21:13.564817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:13.564729 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q"] Jul 24 23:21:13.567534 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:13.567509 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2af5eeed-m5f9q"] Jul 24 23:21:14.767738 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:14.767702 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85"] Jul 24 23:21:14.768190 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:14.767950 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" containerID="cri-o://1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66" gracePeriod=30 Jul 24 23:21:15.299620 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.299580 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12592f82-60c9-4800-a9a8-eb4e43acb7a2" path="/var/lib/kubelet/pods/12592f82-60c9-4800-a9a8-eb4e43acb7a2/volumes" Jul 24 23:21:15.524692 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.524648 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g"] Jul 24 23:21:15.525005 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.524990 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12592f82-60c9-4800-a9a8-eb4e43acb7a2" containerName="revoke-keys" Jul 24 23:21:15.525048 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.525007 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12592f82-60c9-4800-a9a8-eb4e43acb7a2" containerName="revoke-keys" Jul 24 23:21:15.525082 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.525074 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12592f82-60c9-4800-a9a8-eb4e43acb7a2" containerName="revoke-keys" Jul 24 23:21:15.528336 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.528316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.531902 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.531876 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a4bebff3-openshift-default-dockercfg-9l6sh\"" Jul 24 23:21:15.541618 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.541594 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g"] Jul 24 23:21:15.636562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-istio-token\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.636760 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-socket\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.636760 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-credential-socket\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.636760 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-envoy\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.636760 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636745 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-certs\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.636760 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d86824-3b29-460e-af19-f6b830588d44-istio-podinfo\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.637093 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-data\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.637093 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpnk\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-kube-api-access-pgpnk\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.637093 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.636869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d86824-3b29-460e-af19-f6b830588d44-istiod-ca-cert\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.737883 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.737853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-socket\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.737889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-credential-socket\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.737907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-envoy\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.737925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-certs\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.737941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d86824-3b29-460e-af19-f6b830588d44-istio-podinfo\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.737968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-data\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-credential-socket\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738419 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-socket\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738476 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738441 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-certs\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738520 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpnk\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-kube-api-access-pgpnk\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738565 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d86824-3b29-460e-af19-f6b830588d44-istiod-ca-cert\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738621 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-istio-token\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.738621 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.738597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-data\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.739279 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.739252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d86824-3b29-460e-af19-f6b830588d44-istiod-ca-cert\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.740364 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.740338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-envoy\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.740447 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.740433 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d86824-3b29-460e-af19-f6b830588d44-istio-podinfo\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.746517 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.746488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-istio-token\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.746803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.746784 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpnk\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-kube-api-access-pgpnk\") pod \"e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.838320 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.838285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:15.967452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.967378 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g"] Jul 24 23:21:15.969646 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:21:15.969617 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d86824_3b29_460e_af19_f6b830588d44.slice/crio-2b8d495a4bb59f2a6d02bc199d980273707dc576ef0cccc68c6b37eb879d21e6 WatchSource:0}: Error finding container 2b8d495a4bb59f2a6d02bc199d980273707dc576ef0cccc68c6b37eb879d21e6: Status 404 returned error can't find the container with id 2b8d495a4bb59f2a6d02bc199d980273707dc576ef0cccc68c6b37eb879d21e6 Jul 24 23:21:15.971767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.971736 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:21:15.971847 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.971804 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:21:15.971847 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:15.971831 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:21:16.470495 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:16.470458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" event={"ID":"62d86824-3b29-460e-af19-f6b830588d44","Type":"ContainerStarted","Data":"daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7"} Jul 24 23:21:16.470689 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:16.470507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" event={"ID":"62d86824-3b29-460e-af19-f6b830588d44","Type":"ContainerStarted","Data":"2b8d495a4bb59f2a6d02bc199d980273707dc576ef0cccc68c6b37eb879d21e6"} Jul 24 23:21:16.490564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:16.490524 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" podStartSLOduration=1.490510535 podStartE2EDuration="1.490510535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:21:16.489074975 +0000 UTC m=+1735.764044073" watchObservedRunningTime="2026-07-24 23:21:16.490510535 +0000 UTC m=+1735.765479630" Jul 24 23:21:16.838964 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:16.838491 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:17.843746 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:17.843717 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:18.479394 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:18.479368 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:18.480282 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:18.480264 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:20.023266 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.023240 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:21:20.080131 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080101 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istiod-ca-cert\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080139 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-token\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080167 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-data\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-certs\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080218 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftx7h\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-kube-api-access-ftx7h\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080255 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-credential-socket\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080553 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080292 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-socket\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080553 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080314 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-podinfo\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080553 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080345 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-envoy\") pod \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\" (UID: \"01cbec88-dbd5-4e43-92a4-ee8bd594540c\") " Jul 24 23:21:20.080733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:21:20.080733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080646 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:21:20.080733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080664 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:21:20.080937 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.080915 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:21:20.081083 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.081065 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-data" (OuterVolumeSpecName: "istio-data") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:21:20.082651 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.082625 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:21:20.082770 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.082639 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-token" (OuterVolumeSpecName: "istio-token") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:21:20.082933 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.082907 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:21:20.082933 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.082920 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-kube-api-access-ftx7h" (OuterVolumeSpecName: "kube-api-access-ftx7h") pod "01cbec88-dbd5-4e43-92a4-ee8bd594540c" (UID: "01cbec88-dbd5-4e43-92a4-ee8bd594540c"). InnerVolumeSpecName "kube-api-access-ftx7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:21:20.181823 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181794 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.181823 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181818 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.181823 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181827 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.182025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181836 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftx7h\" (UniqueName: \"kubernetes.io/projected/01cbec88-dbd5-4e43-92a4-ee8bd594540c-kube-api-access-ftx7h\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.182025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181845 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.182025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181853 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.182025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181861 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.182025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181869 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.182025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.181878 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbec88-dbd5-4e43-92a4-ee8bd594540c-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:20.490877 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.490790 2562 generic.go:356] "Generic (PLEG): container finished" podID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerID="1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66" exitCode=0 Jul 24 23:21:20.490877 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.490844 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" Jul 24 23:21:20.491100 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.490883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" event={"ID":"01cbec88-dbd5-4e43-92a4-ee8bd594540c","Type":"ContainerDied","Data":"1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66"} Jul 24 23:21:20.491100 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.490954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85" event={"ID":"01cbec88-dbd5-4e43-92a4-ee8bd594540c","Type":"ContainerDied","Data":"653ddb76019992ce49259813257b07998d6bb550f598b34107da4679d46fc505"} Jul 24 23:21:20.491100 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.490990 2562 scope.go:117] "RemoveContainer" containerID="1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66" Jul 24 23:21:20.501792 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.501773 2562 scope.go:117] "RemoveContainer" containerID="7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909" Jul 24 23:21:20.509245 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.509227 2562 scope.go:117] "RemoveContainer" containerID="1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66" Jul 24 23:21:20.509498 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:21:20.509478 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66\": container with ID starting with 1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66 not found: ID does not exist" containerID="1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66" Jul 24 23:21:20.509568 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.509510 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66"} err="failed to get container status \"1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66\": rpc error: code = NotFound desc = could not find container \"1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66\": container with ID starting with 1bf1e1f445d5f6c6f5edb46f50d08e2cca37b5422b7fa3060dd538e30e79bf66 not found: ID does not exist" Jul 24 23:21:20.509568 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.509534 2562 scope.go:117] "RemoveContainer" containerID="7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909" Jul 24 23:21:20.509802 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:21:20.509784 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909\": container with ID starting with 7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909 not found: ID does not exist" containerID="7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909" Jul 24 23:21:20.509851 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.509806 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909"} err="failed to get container status \"7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909\": rpc error: code = NotFound desc = could not find container \"7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909\": container with ID starting with 7b375d9da937b5d653bfe3dd4f63a5426414aae23744a9cf76c1f8df39e3a909 not found: ID does not exist" Jul 24 23:21:20.514660 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.514637 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85"] Jul 24 23:21:20.518866 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:20.518845 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2af5eeed-openshift-default-fbfd9568b-k4n85"] Jul 24 23:21:21.299247 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:21.299217 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" path="/var/lib/kubelet/pods/01cbec88-dbd5-4e43-92a4-ee8bd594540c/volumes" Jul 24 23:21:31.542617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:31.542543 2562 generic.go:356] "Generic (PLEG): container finished" podID="62d86824-3b29-460e-af19-f6b830588d44" containerID="daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7" exitCode=0 Jul 24 23:21:31.543120 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:31.542624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" event={"ID":"62d86824-3b29-460e-af19-f6b830588d44","Type":"ContainerDied","Data":"daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7"} Jul 24 23:21:31.543120 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:31.543079 2562 scope.go:117] "RemoveContainer" containerID="daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7" Jul 24 23:21:31.544100 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:31.544073 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:21:31.544192 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:31.544155 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:21:31.544252 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:31.544203 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:21:32.550734 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:32.550690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" event={"ID":"62d86824-3b29-460e-af19-f6b830588d44","Type":"ContainerStarted","Data":"3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef"} Jul 24 23:21:32.841192 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:32.840733 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:32.841192 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:32.840864 2562 patch_prober.go:28] interesting pod/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 24 23:21:32.841192 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:32.840909 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 24 23:21:33.480221 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:33.480173 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:33.839315 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:33.839272 2562 patch_prober.go:28] interesting pod/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 24 23:21:33.839782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:33.839337 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 24 23:21:34.839600 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:34.839563 2562 patch_prober.go:28] interesting pod/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 24 23:21:34.840201 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:34.840168 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 24 23:21:35.841226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:35.841187 2562 patch_prober.go:28] interesting pod/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 24 23:21:35.841820 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:35.841788 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 24 23:21:36.842100 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:36.842066 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:36.843166 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:36.843143 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:21:42.715356 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.715323 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7"] Jul 24 23:21:42.715732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.715618 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" Jul 24 23:21:42.715732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.715629 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" Jul 24 23:21:42.715732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.715639 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" Jul 24 23:21:42.715732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.715646 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" Jul 24 23:21:42.716198 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.716176 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" Jul 24 23:21:42.716278 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.716255 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01cbec88-dbd5-4e43-92a4-ee8bd594540c" containerName="istio-proxy" Jul 24 23:21:42.721529 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.720393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:42.725359 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.725331 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:21:42.725359 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.725350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h9zqg\"" Jul 24 23:21:42.725887 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.725855 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7"] Jul 24 23:21:42.865693 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.865646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714ee46b-a3a6-48df-bc69-295829165639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7\" (UID: \"714ee46b-a3a6-48df-bc69-295829165639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:42.967119 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.967044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714ee46b-a3a6-48df-bc69-295829165639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7\" (UID: \"714ee46b-a3a6-48df-bc69-295829165639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:42.967642 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:42.967624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714ee46b-a3a6-48df-bc69-295829165639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7\" (UID: \"714ee46b-a3a6-48df-bc69-295829165639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:43.033349 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:43.033315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:43.366796 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:43.366773 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7"] Jul 24 23:21:43.371805 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:21:43.371780 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714ee46b_a3a6_48df_bc69_295829165639.slice/crio-67a14e9411bb2c01964e3c3c19f9ae2104b3a356cf8fe27899096695a4562379 WatchSource:0}: Error finding container 67a14e9411bb2c01964e3c3c19f9ae2104b3a356cf8fe27899096695a4562379: Status 404 returned error can't find the container with id 67a14e9411bb2c01964e3c3c19f9ae2104b3a356cf8fe27899096695a4562379 Jul 24 23:21:43.598171 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:43.598141 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" event={"ID":"714ee46b-a3a6-48df-bc69-295829165639","Type":"ContainerStarted","Data":"67a14e9411bb2c01964e3c3c19f9ae2104b3a356cf8fe27899096695a4562379"} Jul 24 23:21:44.603200 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:44.603160 2562 generic.go:356] "Generic (PLEG): container finished" podID="714ee46b-a3a6-48df-bc69-295829165639" containerID="681f1c76c6fb0a5773ec1b353b756a08a180285f8d488a57f83b54cd06c7bcdb" exitCode=0 Jul 24 23:21:44.603566 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:44.603240 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" event={"ID":"714ee46b-a3a6-48df-bc69-295829165639","Type":"ContainerDied","Data":"681f1c76c6fb0a5773ec1b353b756a08a180285f8d488a57f83b54cd06c7bcdb"} Jul 24 23:21:45.733753 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:45.733730 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:45.894926 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:45.894851 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714ee46b-a3a6-48df-bc69-295829165639-maas-api-service-ca\") pod \"714ee46b-a3a6-48df-bc69-295829165639\" (UID: \"714ee46b-a3a6-48df-bc69-295829165639\") " Jul 24 23:21:45.895199 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:45.895179 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714ee46b-a3a6-48df-bc69-295829165639-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "714ee46b-a3a6-48df-bc69-295829165639" (UID: "714ee46b-a3a6-48df-bc69-295829165639"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:21:45.996389 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:45.996363 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/714ee46b-a3a6-48df-bc69-295829165639-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:21:46.613409 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:46.613382 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" Jul 24 23:21:46.613574 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:46.613417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7" event={"ID":"714ee46b-a3a6-48df-bc69-295829165639","Type":"ContainerDied","Data":"67a14e9411bb2c01964e3c3c19f9ae2104b3a356cf8fe27899096695a4562379"} Jul 24 23:21:46.613574 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:46.613458 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a14e9411bb2c01964e3c3c19f9ae2104b3a356cf8fe27899096695a4562379" Jul 24 23:21:57.795484 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:57.795450 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7"] Jul 24 23:21:57.797690 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:57.797651 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4bebff3-c49b7"] Jul 24 23:21:57.923065 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:57.923028 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g"] Jul 24 23:21:57.923321 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:57.923274 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" containerID="cri-o://3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef" gracePeriod=30 Jul 24 23:21:59.299775 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:21:59.299742 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714ee46b-a3a6-48df-bc69-295829165639" path="/var/lib/kubelet/pods/714ee46b-a3a6-48df-bc69-295829165639/volumes" Jul 24 23:22:03.176040 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.176021 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:22:03.238463 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238396 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d86824-3b29-460e-af19-f6b830588d44-istio-podinfo\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238463 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238432 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-istio-token\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238463 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238448 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d86824-3b29-460e-af19-f6b830588d44-istiod-ca-cert\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238487 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpnk\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-kube-api-access-pgpnk\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238516 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-data\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238542 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-envoy\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238561 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-certs\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.238718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.238612 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-credential-socket\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.239248 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.239212 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:22:03.239451 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.239419 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d86824-3b29-460e-af19-f6b830588d44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:22:03.239609 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.239543 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-socket\") pod \"62d86824-3b29-460e-af19-f6b830588d44\" (UID: \"62d86824-3b29-460e-af19-f6b830588d44\") " Jul 24 23:22:03.239762 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.239740 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:22:03.239953 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.239934 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:22:03.240056 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.240026 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-data" (OuterVolumeSpecName: "istio-data") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:22:03.240375 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.240355 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d86824-3b29-460e-af19-f6b830588d44-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.240474 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.240463 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.240552 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.240542 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.240631 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.240620 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.240734 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.240722 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.241661 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.241627 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-kube-api-access-pgpnk" (OuterVolumeSpecName: "kube-api-access-pgpnk") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "kube-api-access-pgpnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:22:03.241819 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.241791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-istio-token" (OuterVolumeSpecName: "istio-token") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:22:03.246945 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.246803 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:22:03.247033 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.246950 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d86824-3b29-460e-af19-f6b830588d44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62d86824-3b29-460e-af19-f6b830588d44" (UID: "62d86824-3b29-460e-af19-f6b830588d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:22:03.341727 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.341704 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d86824-3b29-460e-af19-f6b830588d44-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.341727 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.341724 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.341903 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.341733 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpnk\" (UniqueName: \"kubernetes.io/projected/62d86824-3b29-460e-af19-f6b830588d44-kube-api-access-pgpnk\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.341903 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.341743 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d86824-3b29-460e-af19-f6b830588d44-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:22:03.686871 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.686830 2562 generic.go:356] "Generic (PLEG): container finished" podID="62d86824-3b29-460e-af19-f6b830588d44" containerID="3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef" exitCode=0 Jul 24 23:22:03.687064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.686895 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" Jul 24 23:22:03.687064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.686918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" event={"ID":"62d86824-3b29-460e-af19-f6b830588d44","Type":"ContainerDied","Data":"3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef"} Jul 24 23:22:03.687064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.686984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g" event={"ID":"62d86824-3b29-460e-af19-f6b830588d44","Type":"ContainerDied","Data":"2b8d495a4bb59f2a6d02bc199d980273707dc576ef0cccc68c6b37eb879d21e6"} Jul 24 23:22:03.687064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.687015 2562 scope.go:117] "RemoveContainer" containerID="3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef" Jul 24 23:22:03.695115 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.695098 2562 scope.go:117] "RemoveContainer" containerID="daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7" Jul 24 23:22:03.702412 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.702394 2562 scope.go:117] "RemoveContainer" containerID="3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef" Jul 24 23:22:03.702691 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:22:03.702647 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef\": container with ID starting with 3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef not found: ID does not exist" containerID="3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef" Jul 24 23:22:03.702746 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.702687 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef"} err="failed to get container status \"3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef\": rpc error: code = NotFound desc = could not find container \"3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef\": container with ID starting with 3ac2a62be32d9d762e9734ce168011e39879b18e9d03a08641a03f6bbfa97cef not found: ID does not exist" Jul 24 23:22:03.702746 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.702707 2562 scope.go:117] "RemoveContainer" containerID="daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7" Jul 24 23:22:03.702921 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:22:03.702905 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7\": container with ID starting with daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7 not found: ID does not exist" containerID="daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7" Jul 24 23:22:03.702971 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.702926 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7"} err="failed to get container status \"daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7\": rpc error: code = NotFound desc = could not find container \"daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7\": container with ID starting with daf57d9514f8f4bf67ad5195b75cbf348869392b82e5714a75e4e5d976162ff7 not found: ID does not exist" Jul 24 23:22:03.705376 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.705354 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g"] Jul 24 23:22:03.711185 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:03.711164 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a4bebff3-openshift-default-7c8fd8f6cb-xbd8g"] Jul 24 23:22:05.299282 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:05.299247 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d86824-3b29-460e-af19-f6b830588d44" path="/var/lib/kubelet/pods/62d86824-3b29-460e-af19-f6b830588d44/volumes" Jul 24 23:22:21.234690 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:21.234638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:22:21.237624 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:22:21.237604 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:23:35.967370 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967340 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr"] Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967619 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714ee46b-a3a6-48df-bc69-295829165639" containerName="revoke-keys" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967629 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="714ee46b-a3a6-48df-bc69-295829165639" containerName="revoke-keys" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967645 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967650 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967662 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967668 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967737 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="714ee46b-a3a6-48df-bc69-295829165639" containerName="revoke-keys" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967746 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" Jul 24 23:23:35.967864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.967753 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d86824-3b29-460e-af19-f6b830588d44" containerName="istio-proxy" Jul 24 23:23:35.970607 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.970585 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:35.973004 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.972986 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9b1bceef-openshift-default-dockercfg-vcbf9\"" Jul 24 23:23:35.992969 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:35.992947 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr"] Jul 24 23:23:36.093183 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-data\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-certs\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093258 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9jf\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-kube-api-access-tw9jf\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-credential-socket\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-podinfo\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d94921f-e749-4f2e-a25b-154c9902b8e2-istiod-ca-cert\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-envoy\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-socket\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.093537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.093476 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-token\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194001 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.193969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9jf\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-kube-api-access-tw9jf\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194001 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-credential-socket\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194223 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-podinfo\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194223 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d94921f-e749-4f2e-a25b-154c9902b8e2-istiod-ca-cert\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194308 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-envoy\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194308 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-socket\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194378 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194316 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-token\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194378 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-data\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194474 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-certs\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194474 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-credential-socket\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194666 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-socket\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194808 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194755 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-certs\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.194912 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.194890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-data\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.195041 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.195023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d94921f-e749-4f2e-a25b-154c9902b8e2-istiod-ca-cert\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.196480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.196458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-envoy\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.196604 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.196587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-podinfo\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.201513 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.201495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-token\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.201726 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.201708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9jf\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-kube-api-access-tw9jf\") pod \"e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.281993 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.281918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:36.418305 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.418280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr"] Jul 24 23:23:36.420647 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:23:36.420622 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d94921f_e749_4f2e_a25b_154c9902b8e2.slice/crio-34d244310c6e37ac89a370d2b403fcbad2881303914647e2d6b091501daf0a5b WatchSource:0}: Error finding container 34d244310c6e37ac89a370d2b403fcbad2881303914647e2d6b091501daf0a5b: Status 404 returned error can't find the container with id 34d244310c6e37ac89a370d2b403fcbad2881303914647e2d6b091501daf0a5b Jul 24 23:23:36.423496 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.423453 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:23:36.423631 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.423548 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:23:36.423631 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:36.423587 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:23:37.084670 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:37.084635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" event={"ID":"8d94921f-e749-4f2e-a25b-154c9902b8e2","Type":"ContainerStarted","Data":"b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21"} Jul 24 23:23:37.084670 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:37.084717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" event={"ID":"8d94921f-e749-4f2e-a25b-154c9902b8e2","Type":"ContainerStarted","Data":"34d244310c6e37ac89a370d2b403fcbad2881303914647e2d6b091501daf0a5b"} Jul 24 23:23:37.105149 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:37.105097 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" podStartSLOduration=2.105083426 podStartE2EDuration="2.105083426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:37.103959332 +0000 UTC m=+1876.378928431" watchObservedRunningTime="2026-07-24 23:23:37.105083426 +0000 UTC m=+1876.380052521" Jul 24 23:23:37.282498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:37.282469 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:37.287440 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:37.287418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:38.090075 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:38.090040 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:38.091075 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:38.091054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:40.025254 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.025215 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz"] Jul 24 23:23:40.028815 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.028793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.031923 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.031890 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9b1bceef\"" Jul 24 23:23:40.032034 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.031990 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9b1bceef-dockercfg-smgs7\"" Jul 24 23:23:40.040622 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.040600 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz"] Jul 24 23:23:40.128990 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.128958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clk7c\" (UniqueName: \"kubernetes.io/projected/c5be9552-83c8-44cd-8f96-1ece38c59733-kube-api-access-clk7c\") pod \"payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.128990 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.128993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be9552-83c8-44cd-8f96-1ece38c59733-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.230407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.230371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clk7c\" (UniqueName: \"kubernetes.io/projected/c5be9552-83c8-44cd-8f96-1ece38c59733-kube-api-access-clk7c\") pod \"payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.230407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.230410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be9552-83c8-44cd-8f96-1ece38c59733-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.231290 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.231265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be9552-83c8-44cd-8f96-1ece38c59733-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.238587 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.238553 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clk7c\" (UniqueName: \"kubernetes.io/projected/c5be9552-83c8-44cd-8f96-1ece38c59733-kube-api-access-clk7c\") pod \"payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.338968 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.338938 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:40.469637 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:40.469613 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz"] Jul 24 23:23:40.472312 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:23:40.472283 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5be9552_83c8_44cd_8f96_1ece38c59733.slice/crio-05954f3c1cb3db4d30336c73875419f2f34cb61b3d46d3157cfd2f3cf3498110 WatchSource:0}: Error finding container 05954f3c1cb3db4d30336c73875419f2f34cb61b3d46d3157cfd2f3cf3498110: Status 404 returned error can't find the container with id 05954f3c1cb3db4d30336c73875419f2f34cb61b3d46d3157cfd2f3cf3498110 Jul 24 23:23:41.106936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:41.106899 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" event={"ID":"c5be9552-83c8-44cd-8f96-1ece38c59733","Type":"ContainerStarted","Data":"a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151"} Jul 24 23:23:41.107390 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:41.106949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" event={"ID":"c5be9552-83c8-44cd-8f96-1ece38c59733","Type":"ContainerStarted","Data":"05954f3c1cb3db4d30336c73875419f2f34cb61b3d46d3157cfd2f3cf3498110"} Jul 24 23:23:41.107390 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:41.107054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:41.124377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:41.124325 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" podStartSLOduration=2.124310203 podStartE2EDuration="2.124310203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:41.123041156 +0000 UTC m=+1880.398010250" watchObservedRunningTime="2026-07-24 23:23:41.124310203 +0000 UTC m=+1880.399279297" Jul 24 23:23:52.113402 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.113373 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:23:52.165988 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.165955 2562 generic.go:356] "Generic (PLEG): container finished" podID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerID="b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21" exitCode=0 Jul 24 23:23:52.166160 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.166034 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" event={"ID":"8d94921f-e749-4f2e-a25b-154c9902b8e2","Type":"ContainerDied","Data":"b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21"} Jul 24 23:23:52.166518 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.166501 2562 scope.go:117] "RemoveContainer" containerID="b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21" Jul 24 23:23:52.167626 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.167602 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:23:52.167711 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.167691 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:23:52.167762 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:52.167732 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:23:53.090234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:53.090188 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:53.172481 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:53.172441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" event={"ID":"8d94921f-e749-4f2e-a25b-154c9902b8e2","Type":"ContainerStarted","Data":"8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906"} Jul 24 23:23:53.282945 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:53.282905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:53.284146 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:53.284120 2562 patch_prober.go:28] interesting pod/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 24 23:23:53.284243 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:53.284173 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 24 23:23:54.282907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:54.282873 2562 patch_prober.go:28] interesting pod/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 24 23:23:54.283293 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:54.282925 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 24 23:23:55.283048 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:55.283011 2562 patch_prober.go:28] interesting pod/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 24 23:23:55.283458 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:55.283068 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 24 23:23:56.282626 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:56.282591 2562 patch_prober.go:28] interesting pod/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 24 23:23:56.282815 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:56.282648 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 24 23:23:57.286546 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:57.286515 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:57.286932 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:57.286880 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:23:57.287476 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:23:57.287456 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:24:01.156118 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.156086 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz"] Jul 24 23:24:01.156539 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.156382 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" podUID="c5be9552-83c8-44cd-8f96-1ece38c59733" containerName="payload-processing" containerID="cri-o://a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151" gracePeriod=30 Jul 24 23:24:01.412545 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.412484 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:24:01.504589 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.504550 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be9552-83c8-44cd-8f96-1ece38c59733-plugins-config-volume\") pod \"c5be9552-83c8-44cd-8f96-1ece38c59733\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " Jul 24 23:24:01.504589 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.504594 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clk7c\" (UniqueName: \"kubernetes.io/projected/c5be9552-83c8-44cd-8f96-1ece38c59733-kube-api-access-clk7c\") pod \"c5be9552-83c8-44cd-8f96-1ece38c59733\" (UID: \"c5be9552-83c8-44cd-8f96-1ece38c59733\") " Jul 24 23:24:01.505001 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.504974 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5be9552-83c8-44cd-8f96-1ece38c59733-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5be9552-83c8-44cd-8f96-1ece38c59733" (UID: "c5be9552-83c8-44cd-8f96-1ece38c59733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:24:01.506682 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.506652 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5be9552-83c8-44cd-8f96-1ece38c59733-kube-api-access-clk7c" (OuterVolumeSpecName: "kube-api-access-clk7c") pod "c5be9552-83c8-44cd-8f96-1ece38c59733" (UID: "c5be9552-83c8-44cd-8f96-1ece38c59733"). InnerVolumeSpecName "kube-api-access-clk7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:24:01.605613 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.605584 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be9552-83c8-44cd-8f96-1ece38c59733-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:01.605613 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:01.605609 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clk7c\" (UniqueName: \"kubernetes.io/projected/c5be9552-83c8-44cd-8f96-1ece38c59733-kube-api-access-clk7c\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:02.213115 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.213083 2562 generic.go:356] "Generic (PLEG): container finished" podID="c5be9552-83c8-44cd-8f96-1ece38c59733" containerID="a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151" exitCode=0 Jul 24 23:24:02.213582 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.213130 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" Jul 24 23:24:02.213582 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.213159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" event={"ID":"c5be9552-83c8-44cd-8f96-1ece38c59733","Type":"ContainerDied","Data":"a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151"} Jul 24 23:24:02.213582 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.213220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz" event={"ID":"c5be9552-83c8-44cd-8f96-1ece38c59733","Type":"ContainerDied","Data":"05954f3c1cb3db4d30336c73875419f2f34cb61b3d46d3157cfd2f3cf3498110"} Jul 24 23:24:02.213582 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.213253 2562 scope.go:117] "RemoveContainer" containerID="a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151" Jul 24 23:24:02.221761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.221740 2562 scope.go:117] "RemoveContainer" containerID="a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151" Jul 24 23:24:02.222018 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:24:02.222000 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151\": container with ID starting with a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151 not found: ID does not exist" containerID="a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151" Jul 24 23:24:02.222081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.222025 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151"} err="failed to get container status \"a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151\": rpc error: code = NotFound desc = could not find container \"a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151\": container with ID starting with a3e0c90d4331dad77ed5f0233b34b0cebcfad071262918c115df5afb9b621151 not found: ID does not exist" Jul 24 23:24:02.237328 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.237300 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz"] Jul 24 23:24:02.242411 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:02.242388 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b1bceef-64868479fd-kxsqz"] Jul 24 23:24:03.299577 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:03.299543 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5be9552-83c8-44cd-8f96-1ece38c59733" path="/var/lib/kubelet/pods/c5be9552-83c8-44cd-8f96-1ece38c59733/volumes" Jul 24 23:24:12.876667 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:12.876636 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr"] Jul 24 23:24:12.877081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:12.876877 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" containerID="cri-o://8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906" gracePeriod=30 Jul 24 23:24:18.123198 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.123170 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:24:18.241281 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-socket\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241281 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241228 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-token\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241281 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241268 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-envoy\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-podinfo\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241338 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-data\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241388 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-certs\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241423 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw9jf\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-kube-api-access-tw9jf\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241470 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d94921f-e749-4f2e-a25b-154c9902b8e2-istiod-ca-cert\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241467 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:24:18.241562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241497 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-credential-socket\") pod \"8d94921f-e749-4f2e-a25b-154c9902b8e2\" (UID: \"8d94921f-e749-4f2e-a25b-154c9902b8e2\") " Jul 24 23:24:18.241985 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241752 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.241985 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241963 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:24:18.242098 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.241978 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:24:18.242098 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.242048 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d94921f-e749-4f2e-a25b-154c9902b8e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:24:18.242297 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.242275 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:24:18.243537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.243511 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:24:18.243662 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.243599 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-kube-api-access-tw9jf" (OuterVolumeSpecName: "kube-api-access-tw9jf") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "kube-api-access-tw9jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:24:18.243915 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.243893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:24:18.244002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.243908 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d94921f-e749-4f2e-a25b-154c9902b8e2" (UID: "8d94921f-e749-4f2e-a25b-154c9902b8e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:24:18.299180 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.299144 2562 generic.go:356] "Generic (PLEG): container finished" podID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerID="8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906" exitCode=0 Jul 24 23:24:18.299342 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.299209 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" Jul 24 23:24:18.299342 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.299227 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" event={"ID":"8d94921f-e749-4f2e-a25b-154c9902b8e2","Type":"ContainerDied","Data":"8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906"} Jul 24 23:24:18.299342 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.299280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr" event={"ID":"8d94921f-e749-4f2e-a25b-154c9902b8e2","Type":"ContainerDied","Data":"34d244310c6e37ac89a370d2b403fcbad2881303914647e2d6b091501daf0a5b"} Jul 24 23:24:18.299342 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.299304 2562 scope.go:117] "RemoveContainer" containerID="8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906" Jul 24 23:24:18.308418 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.308400 2562 scope.go:117] "RemoveContainer" containerID="b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21" Jul 24 23:24:18.316268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.316254 2562 scope.go:117] "RemoveContainer" containerID="8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906" Jul 24 23:24:18.316531 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:24:18.316508 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906\": container with ID starting with 8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906 not found: ID does not exist" containerID="8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906" Jul 24 23:24:18.316617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.316535 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906"} err="failed to get container status \"8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906\": rpc error: code = NotFound desc = could not find container \"8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906\": container with ID starting with 8dfcd389311bb7e171cb0918f9656b9e84ea7b872e8873afd64c516ffa128906 not found: ID does not exist" Jul 24 23:24:18.316617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.316564 2562 scope.go:117] "RemoveContainer" containerID="b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21" Jul 24 23:24:18.316942 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:24:18.316921 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21\": container with ID starting with b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21 not found: ID does not exist" containerID="b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21" Jul 24 23:24:18.317015 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.316948 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21"} err="failed to get container status \"b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21\": rpc error: code = NotFound desc = could not find container \"b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21\": container with ID starting with b65d9390ba633e43bd5ed3ea4086054026d591458ed750127ce8d4e714194c21 not found: ID does not exist" Jul 24 23:24:18.322285 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.322264 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr"] Jul 24 23:24:18.326319 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.326297 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9b1bceef-openshift-default-5695cf7bb4-phgrr"] Jul 24 23:24:18.342570 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342539 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9jf\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-kube-api-access-tw9jf\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342570 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342570 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d94921f-e749-4f2e-a25b-154c9902b8e2-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342570 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342580 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342588 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342597 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342605 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342612 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:18.342767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:18.342620 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d94921f-e749-4f2e-a25b-154c9902b8e2-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:24:19.299387 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:19.299354 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" path="/var/lib/kubelet/pods/8d94921f-e749-4f2e-a25b-154c9902b8e2/volumes" Jul 24 23:24:32.737135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737098 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc"] Jul 24 23:24:32.737602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737538 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5be9552-83c8-44cd-8f96-1ece38c59733" containerName="payload-processing" Jul 24 23:24:32.737602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737555 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5be9552-83c8-44cd-8f96-1ece38c59733" containerName="payload-processing" Jul 24 23:24:32.737602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737584 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" Jul 24 23:24:32.737602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737592 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" Jul 24 23:24:32.737602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737601 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" Jul 24 23:24:32.737892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737611 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" Jul 24 23:24:32.737892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737699 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5be9552-83c8-44cd-8f96-1ece38c59733" containerName="payload-processing" Jul 24 23:24:32.737892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.737713 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" Jul 24 23:24:32.740607 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.740587 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.745408 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.745381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e9d44355-openshift-default-dockercfg-zhd4n\"" Jul 24 23:24:32.773462 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.773437 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc"] Jul 24 23:24:32.860286 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860253 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-socket\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860286 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860287 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-podinfo\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860493 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-credential-socket\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860493 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-certs\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860493 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-envoy\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860493 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de1e18d-816e-4515-9dec-adf9aedc78ac-istiod-ca-cert\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2dh\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-kube-api-access-mf2dh\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-token\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.860700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.860569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-data\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961072 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2dh\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-kube-api-access-mf2dh\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-token\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-data\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-socket\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-podinfo\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-credential-socket\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-certs\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-envoy\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961593 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de1e18d-816e-4515-9dec-adf9aedc78ac-istiod-ca-cert\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961593 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-socket\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.961896 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961869 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-data\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.962018 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-certs\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.962018 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.961994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-credential-socket\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.962235 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.962218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de1e18d-816e-4515-9dec-adf9aedc78ac-istiod-ca-cert\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.964135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.964116 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-envoy\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.964438 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.964418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-podinfo\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.972758 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.972739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-token\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:32.972925 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:32.972907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2dh\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-kube-api-access-mf2dh\") pod \"e2e-mt-e9d44355-openshift-default-845b867848-6wckc\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:33.050725 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.050639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:33.193372 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.193344 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc"] Jul 24 23:24:33.195038 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:24:33.195014 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de1e18d_816e_4515_9dec_adf9aedc78ac.slice/crio-1da30e0fcf69973974e707f35570518d02f60acbea36792938f1399da41cecf5 WatchSource:0}: Error finding container 1da30e0fcf69973974e707f35570518d02f60acbea36792938f1399da41cecf5: Status 404 returned error can't find the container with id 1da30e0fcf69973974e707f35570518d02f60acbea36792938f1399da41cecf5 Jul 24 23:24:33.197131 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.197106 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:24:33.197195 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.197163 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:24:33.197195 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.197191 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:24:33.362956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.362918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" event={"ID":"7de1e18d-816e-4515-9dec-adf9aedc78ac","Type":"ContainerStarted","Data":"9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819"} Jul 24 23:24:33.363157 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.362975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" event={"ID":"7de1e18d-816e-4515-9dec-adf9aedc78ac","Type":"ContainerStarted","Data":"1da30e0fcf69973974e707f35570518d02f60acbea36792938f1399da41cecf5"} Jul 24 23:24:33.394740 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:33.394663 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" podStartSLOduration=1.394644751 podStartE2EDuration="1.394644751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:24:33.392895459 +0000 UTC m=+1932.667864555" watchObservedRunningTime="2026-07-24 23:24:33.394644751 +0000 UTC m=+1932.669613848" Jul 24 23:24:34.050922 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:34.050886 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:34.055739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:34.055713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:34.367977 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:34.367949 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:34.368844 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:34.368824 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:48.430623 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:48.430592 2562 generic.go:356] "Generic (PLEG): container finished" podID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerID="9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819" exitCode=0 Jul 24 23:24:48.431063 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:48.430688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" event={"ID":"7de1e18d-816e-4515-9dec-adf9aedc78ac","Type":"ContainerDied","Data":"9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819"} Jul 24 23:24:48.431117 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:48.431078 2562 scope.go:117] "RemoveContainer" containerID="9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819" Jul 24 23:24:48.432017 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:48.431990 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:24:48.432131 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:48.432071 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:24:48.432200 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:48.432129 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:24:49.368649 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:49.368616 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:49.439272 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:49.439219 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" event={"ID":"7de1e18d-816e-4515-9dec-adf9aedc78ac","Type":"ContainerStarted","Data":"39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e"} Jul 24 23:24:50.051640 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:50.051590 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:50.052722 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:50.052694 2562 patch_prober.go:28] interesting pod/e2e-mt-e9d44355-openshift-default-845b867848-6wckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 24 23:24:50.052843 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:50.052742 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 24 23:24:51.051940 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:51.051896 2562 patch_prober.go:28] interesting pod/e2e-mt-e9d44355-openshift-default-845b867848-6wckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 24 23:24:51.052350 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:51.051961 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 24 23:24:52.051537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:52.051488 2562 patch_prober.go:28] interesting pod/e2e-mt-e9d44355-openshift-default-845b867848-6wckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 24 23:24:52.051773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:52.051557 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 24 23:24:53.051387 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:53.051336 2562 patch_prober.go:28] interesting pod/e2e-mt-e9d44355-openshift-default-845b867848-6wckc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 24 23:24:53.052024 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:53.051409 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 24 23:24:54.055121 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:54.055091 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:54.055561 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:54.055447 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:24:54.056060 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:24:54.056039 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:25:19.449141 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.449104 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh"] Jul 24 23:25:19.449570 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.449440 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d94921f-e749-4f2e-a25b-154c9902b8e2" containerName="istio-proxy" Jul 24 23:25:19.452314 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.452295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:19.455200 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.455171 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:25:19.455315 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.455179 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h9zqg\"" Jul 24 23:25:19.460143 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.460119 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh"] Jul 24 23:25:19.558622 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.558596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53910f56-193d-4c18-bd32-63a2ad9fe1cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh\" (UID: \"53910f56-193d-4c18-bd32-63a2ad9fe1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:19.659556 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.659522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53910f56-193d-4c18-bd32-63a2ad9fe1cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh\" (UID: \"53910f56-193d-4c18-bd32-63a2ad9fe1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:19.660152 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.660132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53910f56-193d-4c18-bd32-63a2ad9fe1cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh\" (UID: \"53910f56-193d-4c18-bd32-63a2ad9fe1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:19.764854 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.764773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:19.894270 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:19.894242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh"] Jul 24 23:25:19.896379 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:25:19.896349 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53910f56_193d_4c18_bd32_63a2ad9fe1cc.slice/crio-3845285af4e99213df540ab0d9daac58f2d8ac23387407b56a3b37443cd63e92 WatchSource:0}: Error finding container 3845285af4e99213df540ab0d9daac58f2d8ac23387407b56a3b37443cd63e92: Status 404 returned error can't find the container with id 3845285af4e99213df540ab0d9daac58f2d8ac23387407b56a3b37443cd63e92 Jul 24 23:25:20.577569 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:20.577475 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" event={"ID":"53910f56-193d-4c18-bd32-63a2ad9fe1cc","Type":"ContainerStarted","Data":"214eb6496e18ea8c66a9cd1f2c0076932c1936bb73f595dc2002341a00eca512"} Jul 24 23:25:20.578021 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:20.577580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" event={"ID":"53910f56-193d-4c18-bd32-63a2ad9fe1cc","Type":"ContainerStarted","Data":"3845285af4e99213df540ab0d9daac58f2d8ac23387407b56a3b37443cd63e92"} Jul 24 23:25:20.593984 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:20.593937 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" podStartSLOduration=1.593924387 podStartE2EDuration="1.593924387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:25:20.592442686 +0000 UTC m=+1979.867411783" watchObservedRunningTime="2026-07-24 23:25:20.593924387 +0000 UTC m=+1979.868893481" Jul 24 23:25:21.583835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:21.583802 2562 generic.go:356] "Generic (PLEG): container finished" podID="53910f56-193d-4c18-bd32-63a2ad9fe1cc" containerID="214eb6496e18ea8c66a9cd1f2c0076932c1936bb73f595dc2002341a00eca512" exitCode=0 Jul 24 23:25:21.584192 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:21.583894 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" event={"ID":"53910f56-193d-4c18-bd32-63a2ad9fe1cc","Type":"ContainerDied","Data":"214eb6496e18ea8c66a9cd1f2c0076932c1936bb73f595dc2002341a00eca512"} Jul 24 23:25:22.713445 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:22.713421 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:22.786051 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:22.786024 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53910f56-193d-4c18-bd32-63a2ad9fe1cc-maas-api-service-ca\") pod \"53910f56-193d-4c18-bd32-63a2ad9fe1cc\" (UID: \"53910f56-193d-4c18-bd32-63a2ad9fe1cc\") " Jul 24 23:25:22.786393 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:22.786368 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53910f56-193d-4c18-bd32-63a2ad9fe1cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53910f56-193d-4c18-bd32-63a2ad9fe1cc" (UID: "53910f56-193d-4c18-bd32-63a2ad9fe1cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:25:22.887388 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:22.887318 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53910f56-193d-4c18-bd32-63a2ad9fe1cc-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:25:23.594347 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:23.594319 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" Jul 24 23:25:23.594347 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:23.594339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh" event={"ID":"53910f56-193d-4c18-bd32-63a2ad9fe1cc","Type":"ContainerDied","Data":"3845285af4e99213df540ab0d9daac58f2d8ac23387407b56a3b37443cd63e92"} Jul 24 23:25:23.594603 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:23.594383 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3845285af4e99213df540ab0d9daac58f2d8ac23387407b56a3b37443cd63e92" Jul 24 23:25:39.523542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:39.523499 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh"] Jul 24 23:25:39.525037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:39.525014 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d44355-6dsqh"] Jul 24 23:25:41.299503 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:25:41.299472 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53910f56-193d-4c18-bd32-63a2ad9fe1cc" path="/var/lib/kubelet/pods/53910f56-193d-4c18-bd32-63a2ad9fe1cc/volumes" Jul 24 23:26:21.329326 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:21.329296 2562 scope.go:117] "RemoveContainer" containerID="dd43a10cacd22021d61225cca8853eb16fe1086a516eb66fd23ae92f69ab6cb9" Jul 24 23:26:31.491033 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:31.490999 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc"] Jul 24 23:26:31.491506 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:31.491218 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" containerID="cri-o://39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e" gracePeriod=30 Jul 24 23:26:36.735896 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.735865 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:26:36.799079 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799006 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de1e18d-816e-4515-9dec-adf9aedc78ac-istiod-ca-cert\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799079 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799042 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-credential-socket\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799079 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799069 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-socket\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799093 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf2dh\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-kube-api-access-mf2dh\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799127 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-envoy\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799154 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-token\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-podinfo\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799240 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-data\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799267 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-certs\") pod \"7de1e18d-816e-4515-9dec-adf9aedc78ac\" (UID: \"7de1e18d-816e-4515-9dec-adf9aedc78ac\") " Jul 24 23:26:36.799354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799341 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:26:36.799739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799407 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:26:36.799739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799455 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de1e18d-816e-4515-9dec-adf9aedc78ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:26:36.799739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799596 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de1e18d-816e-4515-9dec-adf9aedc78ac-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.799739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799589 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:26:36.799739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799614 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.799739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.799650 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.800294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.800270 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:26:36.801915 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.801886 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:26:36.802020 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.801940 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:26:36.802020 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.801944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:26:36.802020 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.801953 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-kube-api-access-mf2dh" (OuterVolumeSpecName: "kube-api-access-mf2dh") pod "7de1e18d-816e-4515-9dec-adf9aedc78ac" (UID: "7de1e18d-816e-4515-9dec-adf9aedc78ac"). InnerVolumeSpecName "kube-api-access-mf2dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:26:36.900901 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.900872 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf2dh\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-kube-api-access-mf2dh\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.900901 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.900898 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.900901 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.900907 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.901102 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.900915 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.901102 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.900923 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.901102 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.900931 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de1e18d-816e-4515-9dec-adf9aedc78ac-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:26:36.919576 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.919546 2562 generic.go:356] "Generic (PLEG): container finished" podID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerID="39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e" exitCode=0 Jul 24 23:26:36.919721 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.919602 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" Jul 24 23:26:36.919721 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.919617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" event={"ID":"7de1e18d-816e-4515-9dec-adf9aedc78ac","Type":"ContainerDied","Data":"39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e"} Jul 24 23:26:36.919721 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.919696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc" event={"ID":"7de1e18d-816e-4515-9dec-adf9aedc78ac","Type":"ContainerDied","Data":"1da30e0fcf69973974e707f35570518d02f60acbea36792938f1399da41cecf5"} Jul 24 23:26:36.919721 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.919719 2562 scope.go:117] "RemoveContainer" containerID="39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e" Jul 24 23:26:36.928206 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.928189 2562 scope.go:117] "RemoveContainer" containerID="9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819" Jul 24 23:26:36.935756 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.935741 2562 scope.go:117] "RemoveContainer" containerID="39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e" Jul 24 23:26:36.936006 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:26:36.935988 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e\": container with ID starting with 39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e not found: ID does not exist" containerID="39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e" Jul 24 23:26:36.936066 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.936013 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e"} err="failed to get container status \"39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e\": rpc error: code = NotFound desc = could not find container \"39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e\": container with ID starting with 39f3c09c124017f4a478ffed56950dacff59839d0b2511046ea41531a4c9297e not found: ID does not exist" Jul 24 23:26:36.936066 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.936030 2562 scope.go:117] "RemoveContainer" containerID="9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819" Jul 24 23:26:36.936285 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:26:36.936268 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819\": container with ID starting with 9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819 not found: ID does not exist" containerID="9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819" Jul 24 23:26:36.936332 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.936289 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819"} err="failed to get container status \"9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819\": rpc error: code = NotFound desc = could not find container \"9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819\": container with ID starting with 9097882d5ca50177be06b1da15cd13bcf382ceee2b17cb874f6fbd2e61de9819 not found: ID does not exist" Jul 24 23:26:36.963334 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.963308 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc"] Jul 24 23:26:36.985903 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:36.985881 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e9d44355-openshift-default-845b867848-6wckc"] Jul 24 23:26:37.299498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:26:37.299469 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" path="/var/lib/kubelet/pods/7de1e18d-816e-4515-9dec-adf9aedc78ac/volumes" Jul 24 23:27:21.257697 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:21.257646 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:27:21.261768 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:21.261736 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:27:21.346341 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:21.346312 2562 scope.go:117] "RemoveContainer" containerID="adf2072c4ed08b4b6cdecc4db5e6e6dceb60b8c6c0a21e4f2ddaf22f6f0cf45c" Jul 24 23:27:28.096018 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.095985 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp"] Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096277 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096287 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096298 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096303 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096312 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53910f56-193d-4c18-bd32-63a2ad9fe1cc" containerName="revoke-keys" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096318 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53910f56-193d-4c18-bd32-63a2ad9fe1cc" containerName="revoke-keys" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096367 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53910f56-193d-4c18-bd32-63a2ad9fe1cc" containerName="revoke-keys" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096377 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" Jul 24 23:27:28.096450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.096384 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de1e18d-816e-4515-9dec-adf9aedc78ac" containerName="istio-proxy" Jul 24 23:27:28.099215 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.099193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.101544 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.101522 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-eb73b6-openshift-default-dockercfg-dfrbx\"" Jul 24 23:27:28.108907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.108882 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp"] Jul 24 23:27:28.126910 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.126876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-socket\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127034 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.126922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-podinfo\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127034 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.126950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1632cda-23a0-4e51-8f06-0c12d02564ff-istiod-ca-cert\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127034 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.126978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44dm\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-kube-api-access-z44dm\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127220 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.127047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-data\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127220 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.127091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-certs\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127220 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.127134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-token\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127220 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.127177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-credential-socket\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.127220 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.127194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-envoy\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228182 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-data\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-certs\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-token\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-credential-socket\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-envoy\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-socket\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-podinfo\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228731 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1632cda-23a0-4e51-8f06-0c12d02564ff-istiod-ca-cert\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228731 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44dm\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-kube-api-access-z44dm\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228731 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228592 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-data\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-credential-socket\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-certs\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.228907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.228884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-socket\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.229234 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.229211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1632cda-23a0-4e51-8f06-0c12d02564ff-istiod-ca-cert\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.230731 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.230710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-envoy\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.231066 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.231044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-podinfo\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.238537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.238516 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-token\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.238655 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.238635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44dm\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-kube-api-access-z44dm\") pod \"e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.410792 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.410705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:28.544938 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.544918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp"] Jul 24 23:27:28.547610 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:27:28.547579 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1632cda_23a0_4e51_8f06_0c12d02564ff.slice/crio-18d87ff5f67e56a5c63ee3e5ec25858739674628e064043a6fad991e8ed19941 WatchSource:0}: Error finding container 18d87ff5f67e56a5c63ee3e5ec25858739674628e064043a6fad991e8ed19941: Status 404 returned error can't find the container with id 18d87ff5f67e56a5c63ee3e5ec25858739674628e064043a6fad991e8ed19941 Jul 24 23:27:28.550274 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.550228 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:28.550395 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.550380 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:28.550482 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:28.550464 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:29.143927 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:29.143893 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" event={"ID":"e1632cda-23a0-4e51-8f06-0c12d02564ff","Type":"ContainerStarted","Data":"ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9"} Jul 24 23:27:29.144325 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:29.143981 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" event={"ID":"e1632cda-23a0-4e51-8f06-0c12d02564ff","Type":"ContainerStarted","Data":"18d87ff5f67e56a5c63ee3e5ec25858739674628e064043a6fad991e8ed19941"} Jul 24 23:27:29.165144 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:29.165093 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podStartSLOduration=1.165076678 podStartE2EDuration="1.165076678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:29.163695096 +0000 UTC m=+2108.438664196" watchObservedRunningTime="2026-07-24 23:27:29.165076678 +0000 UTC m=+2108.440045774" Jul 24 23:27:29.411560 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:29.411475 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:29.416456 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:29.416430 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:30.147566 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:30.147537 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:30.148612 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:30.148594 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:32.009581 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.009543 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw"] Jul 24 23:27:32.012642 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.012623 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.015260 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.015232 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-eb73b6\"" Jul 24 23:27:32.015413 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.015397 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-eb73b6-dockercfg-hf9gc\"" Jul 24 23:27:32.021487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.021465 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw"] Jul 24 23:27:32.056327 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.056301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8xc\" (UniqueName: \"kubernetes.io/projected/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-kube-api-access-sx8xc\") pod \"payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.056451 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.056340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.156922 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.156884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8xc\" (UniqueName: \"kubernetes.io/projected/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-kube-api-access-sx8xc\") pod \"payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.157081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.156935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.157636 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.157615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.165836 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.165812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8xc\" (UniqueName: \"kubernetes.io/projected/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-kube-api-access-sx8xc\") pod \"payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.324884 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.324799 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:32.460302 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:32.460269 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw"] Jul 24 23:27:32.462702 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:27:32.462648 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8936c664_e88d_4dc8_ad2f_ef5c1b06440f.slice/crio-6f4dae281d9561c4bb46032ed43dae9166bede7542b0bb86a8c7ef90a86079fc WatchSource:0}: Error finding container 6f4dae281d9561c4bb46032ed43dae9166bede7542b0bb86a8c7ef90a86079fc: Status 404 returned error can't find the container with id 6f4dae281d9561c4bb46032ed43dae9166bede7542b0bb86a8c7ef90a86079fc Jul 24 23:27:33.162613 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:33.162573 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" event={"ID":"8936c664-e88d-4dc8-ad2f-ef5c1b06440f","Type":"ContainerStarted","Data":"67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c"} Jul 24 23:27:33.163128 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:33.162645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" event={"ID":"8936c664-e88d-4dc8-ad2f-ef5c1b06440f","Type":"ContainerStarted","Data":"6f4dae281d9561c4bb46032ed43dae9166bede7542b0bb86a8c7ef90a86079fc"} Jul 24 23:27:33.163128 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:33.162773 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:33.179502 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:33.179451 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" podStartSLOduration=2.17943629 podStartE2EDuration="2.17943629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:33.178048591 +0000 UTC m=+2112.453017686" watchObservedRunningTime="2026-07-24 23:27:33.17943629 +0000 UTC m=+2112.454405459" Jul 24 23:27:36.219538 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.219507 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5"] Jul 24 23:27:36.223082 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.223064 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.225718 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.225696 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0f3852-openshift-default-dockercfg-hnp6r\"" Jul 24 23:27:36.234784 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.234763 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5"] Jul 24 23:27:36.298146 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-certs\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298146 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istiod-ca-cert\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298345 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-credential-socket\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298345 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-socket\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298345 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-token\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298345 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-data\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298345 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-envoy\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l7w\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-kube-api-access-65l7w\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.298537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.298405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-podinfo\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399124 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l7w\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-kube-api-access-65l7w\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399307 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-podinfo\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399307 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-certs\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399307 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istiod-ca-cert\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399307 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-credential-socket\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399525 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-socket\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399525 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-token\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399525 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-data\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399703 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-envoy\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399703 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399617 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-credential-socket\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399703 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-certs\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399851 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-socket\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.399851 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.399764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-data\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.400224 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.400204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istiod-ca-cert\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.401801 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.401781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-envoy\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.402003 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.401988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-podinfo\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.406817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.406799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-token\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.406976 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.406959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l7w\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-kube-api-access-65l7w\") pod \"e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.535351 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.535268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:36.695349 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.695316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5"] Jul 24 23:27:36.699168 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:27:36.699140 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e308a26_bb6e_478e_9f76_53c3f7ad4772.slice/crio-865e8d2663099b219f4f330228576cbd014252cd241090f937e35bf8daf87c7e WatchSource:0}: Error finding container 865e8d2663099b219f4f330228576cbd014252cd241090f937e35bf8daf87c7e: Status 404 returned error can't find the container with id 865e8d2663099b219f4f330228576cbd014252cd241090f937e35bf8daf87c7e Jul 24 23:27:36.701946 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.701908 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:36.702059 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.701989 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:36.702059 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:36.702020 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:37.181583 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:37.181550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" event={"ID":"4e308a26-bb6e-478e-9f76-53c3f7ad4772","Type":"ContainerStarted","Data":"914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2"} Jul 24 23:27:37.181776 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:37.181605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" event={"ID":"4e308a26-bb6e-478e-9f76-53c3f7ad4772","Type":"ContainerStarted","Data":"865e8d2663099b219f4f330228576cbd014252cd241090f937e35bf8daf87c7e"} Jul 24 23:27:37.203941 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:37.203886 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podStartSLOduration=1.203869625 podStartE2EDuration="1.203869625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:37.201354839 +0000 UTC m=+2116.476323935" watchObservedRunningTime="2026-07-24 23:27:37.203869625 +0000 UTC m=+2116.478838719" Jul 24 23:27:37.535828 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:37.535739 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:37.540562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:37.540538 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:38.186420 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:38.186386 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:38.187436 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:38.187416 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:44.168480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:44.168448 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:27:45.149552 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.149515 2562 patch_prober.go:28] interesting pod/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 24 23:27:45.149781 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.149573 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 24 23:27:45.221468 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.221435 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerID="ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9" exitCode=0 Jul 24 23:27:45.221942 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.221509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" event={"ID":"e1632cda-23a0-4e51-8f06-0c12d02564ff","Type":"ContainerDied","Data":"ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9"} Jul 24 23:27:45.221942 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.221902 2562 scope.go:117] "RemoveContainer" containerID="ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9" Jul 24 23:27:45.222911 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.222888 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:45.222980 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.222953 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:45.223018 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:45.222988 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:46.230944 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:46.230846 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" event={"ID":"e1632cda-23a0-4e51-8f06-0c12d02564ff","Type":"ContainerStarted","Data":"b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef"} Jul 24 23:27:46.411832 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:46.411792 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:46.412036 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:46.411978 2562 patch_prober.go:28] interesting pod/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 24 23:27:46.412111 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:46.412037 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 24 23:27:47.412016 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:47.411977 2562 patch_prober.go:28] interesting pod/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 24 23:27:47.412413 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:47.412031 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 24 23:27:48.411231 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:48.411192 2562 patch_prober.go:28] interesting pod/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 24 23:27:48.411413 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:48.411249 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 24 23:27:49.411398 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:49.411364 2562 patch_prober.go:28] interesting pod/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 24 23:27:49.411814 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:49.411417 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 24 23:27:50.415471 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:50.415442 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:50.415855 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:50.415785 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:50.416490 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:50.416472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:27:53.187617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.187533 2562 patch_prober.go:28] interesting pod/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 24 23:27:53.187617 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.187589 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 24 23:27:53.287014 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.286977 2562 generic.go:356] "Generic (PLEG): container finished" podID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerID="914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2" exitCode=0 Jul 24 23:27:53.287200 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.287058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" event={"ID":"4e308a26-bb6e-478e-9f76-53c3f7ad4772","Type":"ContainerDied","Data":"914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2"} Jul 24 23:27:53.287497 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.287468 2562 scope.go:117] "RemoveContainer" containerID="914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2" Jul 24 23:27:53.288463 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.288438 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:53.288562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.288500 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:53.288562 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:53.288532 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:27:54.293508 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:54.293474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" event={"ID":"4e308a26-bb6e-478e-9f76-53c3f7ad4772","Type":"ContainerStarted","Data":"deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d"} Jul 24 23:27:54.536027 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:54.535990 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:54.536972 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:54.536944 2562 patch_prober.go:28] interesting pod/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 24 23:27:54.537172 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:54.537149 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 24 23:27:55.536716 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:55.536659 2562 patch_prober.go:28] interesting pod/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 24 23:27:55.537276 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:55.537245 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 24 23:27:56.545847 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:56.545804 2562 patch_prober.go:28] interesting pod/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 24 23:27:56.546467 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:56.546428 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 24 23:27:57.536378 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:57.536331 2562 patch_prober.go:28] interesting pod/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 24 23:27:57.536586 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:57.536401 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 24 23:27:58.539605 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:58.539573 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:58.540007 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:58.539972 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:27:58.540796 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:27:58.540773 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:28:21.365336 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:28:21.365306 2562 scope.go:117] "RemoveContainer" containerID="681f1c76c6fb0a5773ec1b353b756a08a180285f8d488a57f83b54cd06c7bcdb" Jul 24 23:29:24.951384 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:24.951322 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8"] Jul 24 23:29:24.954740 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:24.954718 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:24.957262 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:24.957230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h9zqg\"" Jul 24 23:29:24.957386 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:24.957261 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:29:24.962179 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:24.962147 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8"] Jul 24 23:29:25.046868 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:25.046832 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8\" (UID: \"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:25.147287 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:25.147253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8\" (UID: \"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:25.147904 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:25.147884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8\" (UID: \"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:25.266554 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:25.266463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:25.396736 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:25.396709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8"] Jul 24 23:29:25.399783 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:29:25.399744 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec83cc7_2cca_49c0_bc16_3c48fb1f6b92.slice/crio-62f82db0d91fe9f25f5d6bc022f73ebe8e38c948550f222f8cb4db8972942d20 WatchSource:0}: Error finding container 62f82db0d91fe9f25f5d6bc022f73ebe8e38c948550f222f8cb4db8972942d20: Status 404 returned error can't find the container with id 62f82db0d91fe9f25f5d6bc022f73ebe8e38c948550f222f8cb4db8972942d20 Jul 24 23:29:25.688245 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:25.688207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" event={"ID":"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92","Type":"ContainerStarted","Data":"62f82db0d91fe9f25f5d6bc022f73ebe8e38c948550f222f8cb4db8972942d20"} Jul 24 23:29:26.694370 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:26.694334 2562 generic.go:356] "Generic (PLEG): container finished" podID="3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92" containerID="f8ff38cde84cf77e4db8ed5b21355470faf959944696cea9a46023032b71c63f" exitCode=0 Jul 24 23:29:26.694822 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:26.694418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" event={"ID":"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92","Type":"ContainerDied","Data":"f8ff38cde84cf77e4db8ed5b21355470faf959944696cea9a46023032b71c63f"} Jul 24 23:29:27.833252 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:27.833228 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:27.871754 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:27.871723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92-maas-api-service-ca\") pod \"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92\" (UID: \"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92\") " Jul 24 23:29:27.872032 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:27.872010 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92" (UID: "3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:29:27.973313 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:27.973234 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92-maas-api-service-ca\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:28.704414 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:28.704382 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" Jul 24 23:29:28.704601 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:28.704387 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8" event={"ID":"3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92","Type":"ContainerDied","Data":"62f82db0d91fe9f25f5d6bc022f73ebe8e38c948550f222f8cb4db8972942d20"} Jul 24 23:29:28.704601 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:28.704505 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62f82db0d91fe9f25f5d6bc022f73ebe8e38c948550f222f8cb4db8972942d20" Jul 24 23:29:35.378273 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.378236 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw"] Jul 24 23:29:35.378702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.378454 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" podUID="8936c664-e88d-4dc8-ad2f-ef5c1b06440f" containerName="payload-processing" containerID="cri-o://67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c" gracePeriod=30 Jul 24 23:29:35.632000 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.631938 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:29:35.735878 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.735839 2562 generic.go:356] "Generic (PLEG): container finished" podID="8936c664-e88d-4dc8-ad2f-ef5c1b06440f" containerID="67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c" exitCode=0 Jul 24 23:29:35.736037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.735885 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" Jul 24 23:29:35.736037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.735900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" event={"ID":"8936c664-e88d-4dc8-ad2f-ef5c1b06440f","Type":"ContainerDied","Data":"67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c"} Jul 24 23:29:35.736037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.735954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw" event={"ID":"8936c664-e88d-4dc8-ad2f-ef5c1b06440f","Type":"ContainerDied","Data":"6f4dae281d9561c4bb46032ed43dae9166bede7542b0bb86a8c7ef90a86079fc"} Jul 24 23:29:35.736037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.735983 2562 scope.go:117] "RemoveContainer" containerID="67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c" Jul 24 23:29:35.741956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.741928 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-plugins-config-volume\") pod \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " Jul 24 23:29:35.742081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.741966 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx8xc\" (UniqueName: \"kubernetes.io/projected/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-kube-api-access-sx8xc\") pod \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\" (UID: \"8936c664-e88d-4dc8-ad2f-ef5c1b06440f\") " Jul 24 23:29:35.742332 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.742310 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8936c664-e88d-4dc8-ad2f-ef5c1b06440f" (UID: "8936c664-e88d-4dc8-ad2f-ef5c1b06440f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:29:35.744041 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.744020 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-kube-api-access-sx8xc" (OuterVolumeSpecName: "kube-api-access-sx8xc") pod "8936c664-e88d-4dc8-ad2f-ef5c1b06440f" (UID: "8936c664-e88d-4dc8-ad2f-ef5c1b06440f"). InnerVolumeSpecName "kube-api-access-sx8xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:29:35.744387 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.744369 2562 scope.go:117] "RemoveContainer" containerID="67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c" Jul 24 23:29:35.744650 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:29:35.744621 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c\": container with ID starting with 67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c not found: ID does not exist" containerID="67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c" Jul 24 23:29:35.744749 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.744657 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c"} err="failed to get container status \"67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c\": rpc error: code = NotFound desc = could not find container \"67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c\": container with ID starting with 67e797c30147178efd5a715c69e4891730b28ee0fc42ec9737622c8ab14cb03c not found: ID does not exist" Jul 24 23:29:35.842940 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.842914 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:35.842940 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:35.842938 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8xc\" (UniqueName: \"kubernetes.io/projected/8936c664-e88d-4dc8-ad2f-ef5c1b06440f-kube-api-access-sx8xc\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:36.063983 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:36.063955 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw"] Jul 24 23:29:36.069745 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:36.069721 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eb73b6-76f98bf65b-n2kxw"] Jul 24 23:29:37.299374 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:37.299343 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8936c664-e88d-4dc8-ad2f-ef5c1b06440f" path="/var/lib/kubelet/pods/8936c664-e88d-4dc8-ad2f-ef5c1b06440f/volumes" Jul 24 23:29:40.058217 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:40.058181 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8"] Jul 24 23:29:40.062618 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:40.062591 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eb73b6-8vkh8"] Jul 24 23:29:41.300477 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:41.300443 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92" path="/var/lib/kubelet/pods/3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92/volumes" Jul 24 23:29:46.501928 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:46.501895 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp"] Jul 24 23:29:46.502395 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:46.502099 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" containerID="cri-o://b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef" gracePeriod=30 Jul 24 23:29:50.418357 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:50.418317 2562 patch_prober.go:28] interesting pod/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 23:29:50.418743 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:50.418376 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 23:29:51.754120 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.754096 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:29:51.809326 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.809265 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerID="b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef" exitCode=0 Jul 24 23:29:51.809326 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.809298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" event={"ID":"e1632cda-23a0-4e51-8f06-0c12d02564ff","Type":"ContainerDied","Data":"b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef"} Jul 24 23:29:51.809326 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.809311 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" Jul 24 23:29:51.809542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.809334 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp" event={"ID":"e1632cda-23a0-4e51-8f06-0c12d02564ff","Type":"ContainerDied","Data":"18d87ff5f67e56a5c63ee3e5ec25858739674628e064043a6fad991e8ed19941"} Jul 24 23:29:51.809542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.809358 2562 scope.go:117] "RemoveContainer" containerID="b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef" Jul 24 23:29:51.817199 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.817182 2562 scope.go:117] "RemoveContainer" containerID="ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9" Jul 24 23:29:51.824557 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.824543 2562 scope.go:117] "RemoveContainer" containerID="b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef" Jul 24 23:29:51.824829 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:29:51.824811 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef\": container with ID starting with b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef not found: ID does not exist" containerID="b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef" Jul 24 23:29:51.824891 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.824841 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef"} err="failed to get container status \"b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef\": rpc error: code = NotFound desc = could not find container \"b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef\": container with ID starting with b48d4665350a21713afe58a758a76c5474201f0f2520fdfd815888cd9094c7ef not found: ID does not exist" Jul 24 23:29:51.824891 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.824860 2562 scope.go:117] "RemoveContainer" containerID="ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9" Jul 24 23:29:51.825106 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:29:51.825089 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9\": container with ID starting with ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9 not found: ID does not exist" containerID="ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9" Jul 24 23:29:51.825149 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.825113 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9"} err="failed to get container status \"ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9\": rpc error: code = NotFound desc = could not find container \"ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9\": container with ID starting with ddd15b948b90b6d8eb9d5e769d5e455eb57b4a2ebd3629a81b395997ea7897b9 not found: ID does not exist" Jul 24 23:29:51.873899 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.873880 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-data\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.873962 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.873904 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-credential-socket\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.873962 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.873957 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1632cda-23a0-4e51-8f06-0c12d02564ff-istiod-ca-cert\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874042 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.873978 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z44dm\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-kube-api-access-z44dm\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874042 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874004 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-certs\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874042 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874023 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-podinfo\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874203 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874047 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-token\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874203 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-envoy\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874203 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874113 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-socket\") pod \"e1632cda-23a0-4e51-8f06-0c12d02564ff\" (UID: \"e1632cda-23a0-4e51-8f06-0c12d02564ff\") " Jul 24 23:29:51.874351 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874276 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:29:51.874410 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874381 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.874410 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874388 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:29:51.874496 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874449 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1632cda-23a0-4e51-8f06-0c12d02564ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:29:51.874496 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874467 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:29:51.874812 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.874787 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:29:51.876173 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.876145 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-kube-api-access-z44dm" (OuterVolumeSpecName: "kube-api-access-z44dm") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "kube-api-access-z44dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:29:51.876317 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.876300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:29:51.876365 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.876318 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:29:51.876365 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.876338 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1632cda-23a0-4e51-8f06-0c12d02564ff" (UID: "e1632cda-23a0-4e51-8f06-0c12d02564ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:29:51.975568 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975536 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975568 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975567 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975568 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975576 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975751 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975584 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975751 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975593 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1632cda-23a0-4e51-8f06-0c12d02564ff-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975751 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975601 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1632cda-23a0-4e51-8f06-0c12d02564ff-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975751 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975609 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z44dm\" (UniqueName: \"kubernetes.io/projected/e1632cda-23a0-4e51-8f06-0c12d02564ff-kube-api-access-z44dm\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:51.975751 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:51.975617 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1632cda-23a0-4e51-8f06-0c12d02564ff-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:29:52.137693 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:52.137648 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp"] Jul 24 23:29:52.142439 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:52.142414 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-eb73b6-openshift-default-56f8686fdd-w86hp"] Jul 24 23:29:53.299548 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:29:53.299513 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" path="/var/lib/kubelet/pods/e1632cda-23a0-4e51-8f06-0c12d02564ff/volumes" Jul 24 23:30:08.250766 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:08.250735 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5"] Jul 24 23:30:08.251239 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:08.250943 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" containerID="cri-o://deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d" gracePeriod=30 Jul 24 23:30:09.149370 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149333 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5"] Jul 24 23:30:09.149702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149688 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8936c664-e88d-4dc8-ad2f-ef5c1b06440f" containerName="payload-processing" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149704 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8936c664-e88d-4dc8-ad2f-ef5c1b06440f" containerName="payload-processing" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149714 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92" containerName="revoke-keys" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149720 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92" containerName="revoke-keys" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149737 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149742 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149751 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" Jul 24 23:30:09.149773 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149756 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" Jul 24 23:30:09.150170 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149817 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8936c664-e88d-4dc8-ad2f-ef5c1b06440f" containerName="payload-processing" Jul 24 23:30:09.150170 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149824 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec83cc7-2cca-49c0-bc16-3c48fb1f6b92" containerName="revoke-keys" Jul 24 23:30:09.150170 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.149834 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" Jul 24 23:30:09.153058 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.153029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.155804 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.155785 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5d3442-openshift-default-dockercfg-mbdxw\"" Jul 24 23:30:09.167852 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.167829 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5"] Jul 24 23:30:09.207820 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.207793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-credential-socket\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.207932 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.207826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc8w\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-kube-api-access-knc8w\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.207932 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.207856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-envoy\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.207932 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.207916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6feb23-9fd3-446a-a9cc-df1961363a94-istiod-ca-cert\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.208037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.207963 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-podinfo\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.208037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.207988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-certs\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.208037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.208002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-token\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.208137 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.208039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-data\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.208137 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.208078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-socket\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.308954 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.308925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-certs\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.308954 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.308956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-token\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.308980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-data\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-socket\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-credential-socket\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc8w\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-kube-api-access-knc8w\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-envoy\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309146 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6feb23-9fd3-446a-a9cc-df1961363a94-istiod-ca-cert\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-podinfo\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-certs\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-data\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309480 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309445 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-socket\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309889 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-credential-socket\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.309964 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.309944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6feb23-9fd3-446a-a9cc-df1961363a94-istiod-ca-cert\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.311448 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.311425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-envoy\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.311769 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.311750 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-podinfo\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.317116 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.317089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-token\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.317242 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.317224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc8w\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-kube-api-access-knc8w\") pod \"e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.463043 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.462966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:09.607746 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.607640 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5"] Jul 24 23:30:09.610685 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:30:09.610646 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6feb23_9fd3_446a_a9cc_df1961363a94.slice/crio-3eb7516de8ecccc62f96525cfa09c1ef991b5281ba03ebf3f93c593de8ee8306 WatchSource:0}: Error finding container 3eb7516de8ecccc62f96525cfa09c1ef991b5281ba03ebf3f93c593de8ee8306: Status 404 returned error can't find the container with id 3eb7516de8ecccc62f96525cfa09c1ef991b5281ba03ebf3f93c593de8ee8306 Jul 24 23:30:09.613215 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.613187 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:09.613281 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.613246 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:09.613281 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.613274 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:09.891080 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.891046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" event={"ID":"be6feb23-9fd3-446a-a9cc-df1961363a94","Type":"ContainerStarted","Data":"bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d"} Jul 24 23:30:09.891248 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.891105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" event={"ID":"be6feb23-9fd3-446a-a9cc-df1961363a94","Type":"ContainerStarted","Data":"3eb7516de8ecccc62f96525cfa09c1ef991b5281ba03ebf3f93c593de8ee8306"} Jul 24 23:30:09.913214 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:09.913154 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" podStartSLOduration=0.913139517 podStartE2EDuration="913.139517ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:09.910313822 +0000 UTC m=+2269.185282942" watchObservedRunningTime="2026-07-24 23:30:09.913139517 +0000 UTC m=+2269.188108613" Jul 24 23:30:10.463991 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:10.463951 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:10.468907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:10.468883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:10.895381 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:10.895354 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:10.896365 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:10.896347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:13.495850 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.495827 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:30:13.545068 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.544985 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-credential-socket\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545068 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545036 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-podinfo\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545071 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istiod-ca-cert\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-token\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65l7w\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-kube-api-access-65l7w\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545204 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-envoy\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545234 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-certs\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545264 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:13.545602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545273 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-socket\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545381 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-data\") pod \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\" (UID: \"4e308a26-bb6e-478e-9f76-53c3f7ad4772\") " Jul 24 23:30:13.545602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545518 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:13.545602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545539 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:30:13.545829 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545653 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:13.545829 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545741 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.545829 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545761 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.545829 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545776 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.545829 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.545789 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.546377 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.546339 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:13.547424 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.547401 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:13.547513 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.547420 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-kube-api-access-65l7w" (OuterVolumeSpecName: "kube-api-access-65l7w") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "kube-api-access-65l7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:13.547912 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.547891 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:30:13.548016 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.547898 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e308a26-bb6e-478e-9f76-53c3f7ad4772" (UID: "4e308a26-bb6e-478e-9f76-53c3f7ad4772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:13.646766 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.646737 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.646766 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.646761 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.646766 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.646770 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.647058 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.646779 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.647058 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.646789 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l7w\" (UniqueName: \"kubernetes.io/projected/4e308a26-bb6e-478e-9f76-53c3f7ad4772-kube-api-access-65l7w\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:30:13.910670 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.910636 2562 generic.go:356] "Generic (PLEG): container finished" podID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerID="deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d" exitCode=0 Jul 24 23:30:13.910856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.910733 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" Jul 24 23:30:13.910856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.910725 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" event={"ID":"4e308a26-bb6e-478e-9f76-53c3f7ad4772","Type":"ContainerDied","Data":"deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d"} Jul 24 23:30:13.910937 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.910887 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5" event={"ID":"4e308a26-bb6e-478e-9f76-53c3f7ad4772","Type":"ContainerDied","Data":"865e8d2663099b219f4f330228576cbd014252cd241090f937e35bf8daf87c7e"} Jul 24 23:30:13.910937 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.910915 2562 scope.go:117] "RemoveContainer" containerID="deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d" Jul 24 23:30:13.920122 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.920104 2562 scope.go:117] "RemoveContainer" containerID="914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2" Jul 24 23:30:13.927993 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.927971 2562 scope.go:117] "RemoveContainer" containerID="deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d" Jul 24 23:30:13.928268 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:30:13.928246 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d\": container with ID starting with deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d not found: ID does not exist" containerID="deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d" Jul 24 23:30:13.928366 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.928274 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d"} err="failed to get container status \"deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d\": rpc error: code = NotFound desc = could not find container \"deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d\": container with ID starting with deab35a44cbcf0344de0a02d68132840e682db63b801335ee8db04fa5a7fbc2d not found: ID does not exist" Jul 24 23:30:13.928366 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.928291 2562 scope.go:117] "RemoveContainer" containerID="914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2" Jul 24 23:30:13.928537 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:30:13.928518 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2\": container with ID starting with 914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2 not found: ID does not exist" containerID="914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2" Jul 24 23:30:13.928574 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.928541 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2"} err="failed to get container status \"914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2\": rpc error: code = NotFound desc = could not find container \"914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2\": container with ID starting with 914c84f94e94b5b8dd4b20af22dcdd723477717750cfe0074591a4772e4f55c2 not found: ID does not exist" Jul 24 23:30:13.933812 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.933789 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5"] Jul 24 23:30:13.939491 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:13.939468 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0f3852-openshift-default-86b6fc6fb-7c5g5"] Jul 24 23:30:15.299577 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:15.299544 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" path="/var/lib/kubelet/pods/4e308a26-bb6e-478e-9f76-53c3f7ad4772/volumes" Jul 24 23:30:17.145700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145129 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj"] Jul 24 23:30:17.145700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145670 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" Jul 24 23:30:17.145700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145706 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" Jul 24 23:30:17.146294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145722 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" Jul 24 23:30:17.146294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145732 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" Jul 24 23:30:17.146294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145805 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" Jul 24 23:30:17.146294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145819 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1632cda-23a0-4e51-8f06-0c12d02564ff" containerName="istio-proxy" Jul 24 23:30:17.146294 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.145829 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e308a26-bb6e-478e-9f76-53c3f7ad4772" containerName="istio-proxy" Jul 24 23:30:17.157795 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.157296 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj"] Jul 24 23:30:17.157795 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.157417 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.160148 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.160085 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-183bb6-openshift-default-dockercfg-cfsnd\"" Jul 24 23:30:17.175569 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30197fd5-0e3c-4c15-9536-621208fef204-istio-podinfo\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.175788 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58kc\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-kube-api-access-l58kc\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.175788 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-envoy\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.175788 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-socket\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.175788 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-certs\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.176182 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30197fd5-0e3c-4c15-9536-621208fef204-istiod-ca-cert\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.176182 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-credential-socket\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.176182 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-data\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.176182 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.175918 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-istio-token\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277007 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.276967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-socket\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277007 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-certs\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30197fd5-0e3c-4c15-9536-621208fef204-istiod-ca-cert\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-credential-socket\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-data\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-istio-token\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30197fd5-0e3c-4c15-9536-621208fef204-istio-podinfo\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58kc\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-kube-api-access-l58kc\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277265 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-envoy\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277619 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-socket\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277619 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277467 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-certs\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-credential-socket\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277832 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30197fd5-0e3c-4c15-9536-621208fef204-istiod-ca-cert\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.277873 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.277851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-data\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.279811 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.279774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30197fd5-0e3c-4c15-9536-621208fef204-istio-podinfo\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.280222 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.280202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-envoy\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.286510 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.286480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-istio-token\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.286625 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.286554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58kc\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-kube-api-access-l58kc\") pod \"e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.470762 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.470639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:17.652138 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.652099 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj"] Jul 24 23:30:17.656635 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:30:17.656608 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30197fd5_0e3c_4c15_9536_621208fef204.slice/crio-701331dc62e4c0966bc40f89b57384b3ffc00f39d1213e31fc58cd73a89bb66f WatchSource:0}: Error finding container 701331dc62e4c0966bc40f89b57384b3ffc00f39d1213e31fc58cd73a89bb66f: Status 404 returned error can't find the container with id 701331dc62e4c0966bc40f89b57384b3ffc00f39d1213e31fc58cd73a89bb66f Jul 24 23:30:17.660112 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.660073 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:17.660189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.660163 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:17.660250 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.660205 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:17.962919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.962879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" event={"ID":"30197fd5-0e3c-4c15-9536-621208fef204","Type":"ContainerStarted","Data":"34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d"} Jul 24 23:30:17.963196 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.963174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" event={"ID":"30197fd5-0e3c-4c15-9536-621208fef204","Type":"ContainerStarted","Data":"701331dc62e4c0966bc40f89b57384b3ffc00f39d1213e31fc58cd73a89bb66f"} Jul 24 23:30:17.980666 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:17.980593 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podStartSLOduration=0.980580435 podStartE2EDuration="980.580435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:17.979727588 +0000 UTC m=+2277.254696688" watchObservedRunningTime="2026-07-24 23:30:17.980580435 +0000 UTC m=+2277.255549531" Jul 24 23:30:18.470907 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:18.470867 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:18.476040 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:18.476014 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:18.967803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:18.967770 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:18.968871 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:18.968853 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:21.025318 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.025284 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr"] Jul 24 23:30:21.028916 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.028890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.034301 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.034248 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-183bb6\"" Jul 24 23:30:21.034477 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.034363 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-183bb6-dockercfg-qghxm\"" Jul 24 23:30:21.047213 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.047184 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr"] Jul 24 23:30:21.112202 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.112165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95feb78-d428-453c-9587-0bbd8bba3bce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.112482 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.112461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52shb\" (UniqueName: \"kubernetes.io/projected/e95feb78-d428-453c-9587-0bbd8bba3bce-kube-api-access-52shb\") pod \"payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.213224 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.213183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95feb78-d428-453c-9587-0bbd8bba3bce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.213224 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.213239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52shb\" (UniqueName: \"kubernetes.io/projected/e95feb78-d428-453c-9587-0bbd8bba3bce-kube-api-access-52shb\") pod \"payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.215783 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.215765 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-183bb6\"" Jul 24 23:30:21.221960 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.221943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52shb\" (UniqueName: \"kubernetes.io/projected/e95feb78-d428-453c-9587-0bbd8bba3bce-kube-api-access-52shb\") pod \"payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.224189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.224164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95feb78-d428-453c-9587-0bbd8bba3bce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.342612 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.342579 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-183bb6-dockercfg-qghxm\"" Jul 24 23:30:21.350860 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.350822 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:21.506086 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.506054 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr"] Jul 24 23:30:21.509363 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:30:21.509337 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95feb78_d428_453c_9587_0bbd8bba3bce.slice/crio-f44a5274fbbed0f05a7bd816d0f5a6fecc3320f81ef5a7249b6a7faeb0061871 WatchSource:0}: Error finding container f44a5274fbbed0f05a7bd816d0f5a6fecc3320f81ef5a7249b6a7faeb0061871: Status 404 returned error can't find the container with id f44a5274fbbed0f05a7bd816d0f5a6fecc3320f81ef5a7249b6a7faeb0061871 Jul 24 23:30:21.983385 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.983353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" event={"ID":"e95feb78-d428-453c-9587-0bbd8bba3bce","Type":"ContainerStarted","Data":"3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f"} Jul 24 23:30:21.983567 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.983409 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" event={"ID":"e95feb78-d428-453c-9587-0bbd8bba3bce","Type":"ContainerStarted","Data":"f44a5274fbbed0f05a7bd816d0f5a6fecc3320f81ef5a7249b6a7faeb0061871"} Jul 24 23:30:21.983567 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:21.983434 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:22.001052 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.000990 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" podStartSLOduration=1.000958169 podStartE2EDuration="1.000958169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:21.998696786 +0000 UTC m=+2281.273665876" watchObservedRunningTime="2026-07-24 23:30:22.000958169 +0000 UTC m=+2281.275927301" Jul 24 23:30:22.989198 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.989167 2562 generic.go:356] "Generic (PLEG): container finished" podID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerID="bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d" exitCode=0 Jul 24 23:30:22.989595 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.989247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" event={"ID":"be6feb23-9fd3-446a-a9cc-df1961363a94","Type":"ContainerDied","Data":"bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d"} Jul 24 23:30:22.989795 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.989776 2562 scope.go:117] "RemoveContainer" containerID="bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d" Jul 24 23:30:22.990772 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.990747 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:22.990835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.990809 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:22.990875 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:22.990839 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:23.995549 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:23.995516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" event={"ID":"be6feb23-9fd3-446a-a9cc-df1961363a94","Type":"ContainerStarted","Data":"7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c"} Jul 24 23:30:24.463867 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:24.463823 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:24.465106 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:24.465083 2562 patch_prober.go:28] interesting pod/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 23:30:24.465226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:24.465129 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 23:30:25.464092 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:25.464055 2562 patch_prober.go:28] interesting pod/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 23:30:25.464501 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:25.464110 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 23:30:25.895921 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:25.895885 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:26.464137 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:26.464100 2562 patch_prober.go:28] interesting pod/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 23:30:26.464533 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:26.464157 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 23:30:27.464848 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:27.464813 2562 patch_prober.go:28] interesting pod/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 24 23:30:27.465256 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:27.464965 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 24 23:30:28.467007 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:28.466976 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:28.468034 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:28.468017 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:30:32.991255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:32.991221 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:30:35.008908 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:35.008850 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 23:30:35.009432 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:35.008929 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 23:30:36.064507 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:36.064405 2562 generic.go:356] "Generic (PLEG): container finished" podID="30197fd5-0e3c-4c15-9536-621208fef204" containerID="34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d" exitCode=0 Jul 24 23:30:36.065014 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:36.064500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" event={"ID":"30197fd5-0e3c-4c15-9536-621208fef204","Type":"ContainerDied","Data":"34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d"} Jul 24 23:30:36.065014 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:36.064933 2562 scope.go:117] "RemoveContainer" containerID="34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d" Jul 24 23:30:36.065994 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:36.065968 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:36.066057 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:36.066036 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:36.066098 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:36.066067 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:30:37.071093 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:37.071058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" event={"ID":"30197fd5-0e3c-4c15-9536-621208fef204","Type":"ContainerStarted","Data":"4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede"} Jul 24 23:30:37.471532 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:37.471488 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:37.471788 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:37.471766 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:30:37.471843 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:37.471814 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 24 23:30:38.471270 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:38.471231 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:30:38.471737 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:38.471291 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 24 23:30:39.471064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:39.471026 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:30:39.471250 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:39.471094 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 24 23:30:40.471835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:40.471797 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:30:40.472232 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:40.471850 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 24 23:30:41.471883 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:41.471845 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:30:41.472277 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:41.471902 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 24 23:30:42.475493 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:42.475462 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:42.475962 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:42.475737 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:42.476441 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:42.476420 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:30:59.093235 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:59.093198 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5"] Jul 24 23:30:59.093781 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:30:59.093457 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" containerID="cri-o://7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c" gracePeriod=30 Jul 24 23:31:04.344487 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.344460 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:31:04.413956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.413874 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-token\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.413956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.413914 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-podinfo\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.413956 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.413953 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-envoy\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.413979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-socket\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414038 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6feb23-9fd3-446a-a9cc-df1961363a94-istiod-ca-cert\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414078 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-data\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414106 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-credential-socket\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414155 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knc8w\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-kube-api-access-knc8w\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414226 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414189 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-certs\") pod \"be6feb23-9fd3-446a-a9cc-df1961363a94\" (UID: \"be6feb23-9fd3-446a-a9cc-df1961363a94\") " Jul 24 23:31:04.414513 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414361 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:04.414513 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414508 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.414629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414510 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:04.414629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414550 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6feb23-9fd3-446a-a9cc-df1961363a94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:31:04.414629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.414598 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:04.415182 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.415158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-data" (OuterVolumeSpecName: "istio-data") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:04.416204 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.416181 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:31:04.416348 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.416332 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-kube-api-access-knc8w" (OuterVolumeSpecName: "kube-api-access-knc8w") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "kube-api-access-knc8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:31:04.416521 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.416499 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-token" (OuterVolumeSpecName: "istio-token") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:31:04.416608 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.416545 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be6feb23-9fd3-446a-a9cc-df1961363a94" (UID: "be6feb23-9fd3-446a-a9cc-df1961363a94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:04.515443 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515417 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515443 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515444 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515457 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515469 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be6feb23-9fd3-446a-a9cc-df1961363a94-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515479 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515489 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515502 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knc8w\" (UniqueName: \"kubernetes.io/projected/be6feb23-9fd3-446a-a9cc-df1961363a94-kube-api-access-knc8w\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:04.515629 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:04.515514 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6feb23-9fd3-446a-a9cc-df1961363a94-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:05.202627 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.202595 2562 generic.go:356] "Generic (PLEG): container finished" podID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerID="7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c" exitCode=0 Jul 24 23:31:05.202817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.202642 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" Jul 24 23:31:05.202817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.202646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" event={"ID":"be6feb23-9fd3-446a-a9cc-df1961363a94","Type":"ContainerDied","Data":"7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c"} Jul 24 23:31:05.202817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.202704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5" event={"ID":"be6feb23-9fd3-446a-a9cc-df1961363a94","Type":"ContainerDied","Data":"3eb7516de8ecccc62f96525cfa09c1ef991b5281ba03ebf3f93c593de8ee8306"} Jul 24 23:31:05.202817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.202726 2562 scope.go:117] "RemoveContainer" containerID="7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c" Jul 24 23:31:05.211769 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.211597 2562 scope.go:117] "RemoveContainer" containerID="bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d" Jul 24 23:31:05.219069 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.219053 2562 scope.go:117] "RemoveContainer" containerID="7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c" Jul 24 23:31:05.219315 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:31:05.219294 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c\": container with ID starting with 7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c not found: ID does not exist" containerID="7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c" Jul 24 23:31:05.219406 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.219326 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c"} err="failed to get container status \"7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c\": rpc error: code = NotFound desc = could not find container \"7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c\": container with ID starting with 7d5c2b177a6eb2e35896ff027dcd8d277c3b36edae59b2000e8643d06d01fb5c not found: ID does not exist" Jul 24 23:31:05.219406 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.219349 2562 scope.go:117] "RemoveContainer" containerID="bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d" Jul 24 23:31:05.219598 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:31:05.219581 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d\": container with ID starting with bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d not found: ID does not exist" containerID="bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d" Jul 24 23:31:05.219637 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.219604 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d"} err="failed to get container status \"bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d\": rpc error: code = NotFound desc = could not find container \"bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d\": container with ID starting with bb4935829332d3c64b1c433b2e4a8f63dc882322f44e70581b15666981760a7d not found: ID does not exist" Jul 24 23:31:05.225824 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.225802 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5"] Jul 24 23:31:05.229640 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.229618 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5d3442-openshift-default-6f8b89d996-bpfb5"] Jul 24 23:31:05.299240 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:05.299209 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" path="/var/lib/kubelet/pods/be6feb23-9fd3-446a-a9cc-df1961363a94/volumes" Jul 24 23:31:15.220194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.220158 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr"] Jul 24 23:31:15.220780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.220430 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" podUID="e95feb78-d428-453c-9587-0bbd8bba3bce" containerName="payload-processing" containerID="cri-o://3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f" gracePeriod=30 Jul 24 23:31:15.481199 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.481131 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:31:15.603437 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.603385 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95feb78-d428-453c-9587-0bbd8bba3bce-plugins-config-volume\") pod \"e95feb78-d428-453c-9587-0bbd8bba3bce\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " Jul 24 23:31:15.603647 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.603477 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52shb\" (UniqueName: \"kubernetes.io/projected/e95feb78-d428-453c-9587-0bbd8bba3bce-kube-api-access-52shb\") pod \"e95feb78-d428-453c-9587-0bbd8bba3bce\" (UID: \"e95feb78-d428-453c-9587-0bbd8bba3bce\") " Jul 24 23:31:15.603821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.603798 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95feb78-d428-453c-9587-0bbd8bba3bce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e95feb78-d428-453c-9587-0bbd8bba3bce" (UID: "e95feb78-d428-453c-9587-0bbd8bba3bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:31:15.605531 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.605501 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95feb78-d428-453c-9587-0bbd8bba3bce-kube-api-access-52shb" (OuterVolumeSpecName: "kube-api-access-52shb") pod "e95feb78-d428-453c-9587-0bbd8bba3bce" (UID: "e95feb78-d428-453c-9587-0bbd8bba3bce"). InnerVolumeSpecName "kube-api-access-52shb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:31:15.704121 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.704068 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95feb78-d428-453c-9587-0bbd8bba3bce-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:15.704121 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:15.704111 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52shb\" (UniqueName: \"kubernetes.io/projected/e95feb78-d428-453c-9587-0bbd8bba3bce-kube-api-access-52shb\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:16.253235 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.253198 2562 generic.go:356] "Generic (PLEG): container finished" podID="e95feb78-d428-453c-9587-0bbd8bba3bce" containerID="3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f" exitCode=0 Jul 24 23:31:16.253733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.253250 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" Jul 24 23:31:16.253733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.253290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" event={"ID":"e95feb78-d428-453c-9587-0bbd8bba3bce","Type":"ContainerDied","Data":"3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f"} Jul 24 23:31:16.253733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.253348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr" event={"ID":"e95feb78-d428-453c-9587-0bbd8bba3bce","Type":"ContainerDied","Data":"f44a5274fbbed0f05a7bd816d0f5a6fecc3320f81ef5a7249b6a7faeb0061871"} Jul 24 23:31:16.253733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.253370 2562 scope.go:117] "RemoveContainer" containerID="3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f" Jul 24 23:31:16.262849 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.262756 2562 scope.go:117] "RemoveContainer" containerID="3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f" Jul 24 23:31:16.263068 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:31:16.263042 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f\": container with ID starting with 3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f not found: ID does not exist" containerID="3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f" Jul 24 23:31:16.263171 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.263074 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f"} err="failed to get container status \"3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f\": rpc error: code = NotFound desc = could not find container \"3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f\": container with ID starting with 3c114040121211058d804f91e3dfffff463c64207604a68c111255654214300f not found: ID does not exist" Jul 24 23:31:16.275041 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.275004 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr"] Jul 24 23:31:16.278847 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:16.278819 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-183bb6-5f76c6f5f8-kckxr"] Jul 24 23:31:17.299995 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:17.299959 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95feb78-d428-453c-9587-0bbd8bba3bce" path="/var/lib/kubelet/pods/e95feb78-d428-453c-9587-0bbd8bba3bce/volumes" Jul 24 23:31:21.413468 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:21.413438 2562 scope.go:117] "RemoveContainer" containerID="214eb6496e18ea8c66a9cd1f2c0076932c1936bb73f595dc2002341a00eca512" Jul 24 23:31:26.315843 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:26.315804 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj"] Jul 24 23:31:26.316275 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:26.316006 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" containerID="cri-o://4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede" gracePeriod=30 Jul 24 23:31:27.212726 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.212663 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc"] Jul 24 23:31:27.213183 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213168 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" Jul 24 23:31:27.213243 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213187 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" Jul 24 23:31:27.213243 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213227 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95feb78-d428-453c-9587-0bbd8bba3bce" containerName="payload-processing" Jul 24 23:31:27.213243 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213237 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95feb78-d428-453c-9587-0bbd8bba3bce" containerName="payload-processing" Jul 24 23:31:27.213350 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213312 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" Jul 24 23:31:27.213350 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213323 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" Jul 24 23:31:27.213350 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.213335 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95feb78-d428-453c-9587-0bbd8bba3bce" containerName="payload-processing" Jul 24 23:31:27.216436 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.216415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.218843 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.218819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fe475f-openshift-default-dockercfg-zvhtg\"" Jul 24 23:31:27.227333 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.227308 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc"] Jul 24 23:31:27.304018 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.303972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-credential-socket\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfht\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-kube-api-access-6cfht\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-socket\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-envoy\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-data\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78faec82-39d5-478b-bb74-31f2f191e523-istio-podinfo\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304559 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-istio-token\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304559 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-certs\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.304559 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.304344 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78faec82-39d5-478b-bb74-31f2f191e523-istiod-ca-cert\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.404836 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.404803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-data\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.404836 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.404844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78faec82-39d5-478b-bb74-31f2f191e523-istio-podinfo\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.404965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-istio-token\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-certs\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78faec82-39d5-478b-bb74-31f2f191e523-istiod-ca-cert\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-credential-socket\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfht\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-kube-api-access-6cfht\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-socket\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405338 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-data\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405752 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-certs\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405752 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405467 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-credential-socket\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405752 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-envoy\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405752 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405705 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-socket\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.405936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.405836 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78faec82-39d5-478b-bb74-31f2f191e523-istiod-ca-cert\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.407543 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.407521 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78faec82-39d5-478b-bb74-31f2f191e523-istio-podinfo\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.407655 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.407522 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-envoy\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.412719 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.412696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfht\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-kube-api-access-6cfht\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.412823 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.412806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-istio-token\") pod \"e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.478288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.478203 2562 patch_prober.go:28] interesting pod/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 23:31:27.478288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.478260 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 23:31:27.528149 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.528114 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:27.666906 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.666836 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc"] Jul 24 23:31:27.669494 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:31:27.669463 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78faec82_39d5_478b_bb74_31f2f191e523.slice/crio-b0a45f58af650c09c907094ec1e617b3968d600097f79fea970b1b4df70b93f5 WatchSource:0}: Error finding container b0a45f58af650c09c907094ec1e617b3968d600097f79fea970b1b4df70b93f5: Status 404 returned error can't find the container with id b0a45f58af650c09c907094ec1e617b3968d600097f79fea970b1b4df70b93f5 Jul 24 23:31:27.671576 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.671544 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:27.671682 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.671608 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:27.671682 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:27.671636 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:28.312969 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:28.312930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" event={"ID":"78faec82-39d5-478b-bb74-31f2f191e523","Type":"ContainerStarted","Data":"5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407"} Jul 24 23:31:28.313202 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:28.312992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" event={"ID":"78faec82-39d5-478b-bb74-31f2f191e523","Type":"ContainerStarted","Data":"b0a45f58af650c09c907094ec1e617b3968d600097f79fea970b1b4df70b93f5"} Jul 24 23:31:28.334464 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:28.334407 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podStartSLOduration=1.3343927070000001 podStartE2EDuration="1.334392707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:31:28.332781473 +0000 UTC m=+2347.607750612" watchObservedRunningTime="2026-07-24 23:31:28.334392707 +0000 UTC m=+2347.609361802" Jul 24 23:31:28.529335 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:28.529293 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:28.534117 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:28.534091 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:29.317085 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:29.317056 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:29.318092 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:29.318069 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:31.575533 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.575473 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:31:31.641471 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641437 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-socket\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641471 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641479 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30197fd5-0e3c-4c15-9536-621208fef204-istiod-ca-cert\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-istio-token\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641523 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-envoy\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641644 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-credential-socket\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641706 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-data\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641941 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641754 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-certs\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641941 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:31.641941 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641801 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l58kc\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-kube-api-access-l58kc\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641941 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641878 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30197fd5-0e3c-4c15-9536-621208fef204-istio-podinfo\") pod \"30197fd5-0e3c-4c15-9536-621208fef204\" (UID: \"30197fd5-0e3c-4c15-9536-621208fef204\") " Jul 24 23:31:31.641941 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641938 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30197fd5-0e3c-4c15-9536-621208fef204-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:31:31.642249 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.641944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:31.642249 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.642190 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.642249 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.642210 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30197fd5-0e3c-4c15-9536-621208fef204-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.642249 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.642226 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.642249 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.642187 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:31.642590 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.642570 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-data" (OuterVolumeSpecName: "istio-data") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:31.643966 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.643938 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-kube-api-access-l58kc" (OuterVolumeSpecName: "kube-api-access-l58kc") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "kube-api-access-l58kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:31:31.644074 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.643944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30197fd5-0e3c-4c15-9536-621208fef204-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:31:31.644141 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.644126 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-istio-token" (OuterVolumeSpecName: "istio-token") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:31:31.644197 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.644188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30197fd5-0e3c-4c15-9536-621208fef204" (UID: "30197fd5-0e3c-4c15-9536-621208fef204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:31.742835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.742793 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30197fd5-0e3c-4c15-9536-621208fef204-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.742835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.742826 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.742835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.742839 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.743086 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.742849 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.743086 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.742861 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30197fd5-0e3c-4c15-9536-621208fef204-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:31.743086 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:31.742872 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58kc\" (UniqueName: \"kubernetes.io/projected/30197fd5-0e3c-4c15-9536-621208fef204-kube-api-access-l58kc\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:31:32.332189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.332152 2562 generic.go:356] "Generic (PLEG): container finished" podID="30197fd5-0e3c-4c15-9536-621208fef204" containerID="4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede" exitCode=0 Jul 24 23:31:32.332372 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.332209 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" Jul 24 23:31:32.332372 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.332245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" event={"ID":"30197fd5-0e3c-4c15-9536-621208fef204","Type":"ContainerDied","Data":"4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede"} Jul 24 23:31:32.332372 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.332302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj" event={"ID":"30197fd5-0e3c-4c15-9536-621208fef204","Type":"ContainerDied","Data":"701331dc62e4c0966bc40f89b57384b3ffc00f39d1213e31fc58cd73a89bb66f"} Jul 24 23:31:32.332372 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.332326 2562 scope.go:117] "RemoveContainer" containerID="4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede" Jul 24 23:31:32.341724 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.341705 2562 scope.go:117] "RemoveContainer" containerID="34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d" Jul 24 23:31:32.349969 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.349950 2562 scope.go:117] "RemoveContainer" containerID="4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede" Jul 24 23:31:32.350240 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:31:32.350222 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede\": container with ID starting with 4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede not found: ID does not exist" containerID="4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede" Jul 24 23:31:32.350295 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.350249 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede"} err="failed to get container status \"4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede\": rpc error: code = NotFound desc = could not find container \"4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede\": container with ID starting with 4e8f0c7e87add1763e558845c10700278f8ae531a5a03eed8f00407fe8da1ede not found: ID does not exist" Jul 24 23:31:32.350295 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.350264 2562 scope.go:117] "RemoveContainer" containerID="34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d" Jul 24 23:31:32.350456 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:31:32.350440 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d\": container with ID starting with 34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d not found: ID does not exist" containerID="34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d" Jul 24 23:31:32.350521 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.350458 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d"} err="failed to get container status \"34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d\": rpc error: code = NotFound desc = could not find container \"34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d\": container with ID starting with 34ff2455602ee21bab75b73921e1f03c6753d0a02efe925cdcbe3b89b01dc79d not found: ID does not exist" Jul 24 23:31:32.357233 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.357211 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj"] Jul 24 23:31:32.364366 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:32.364342 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-183bb6-openshift-default-7b77dd84c4-pw8wj"] Jul 24 23:31:33.300639 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:33.300604 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30197fd5-0e3c-4c15-9536-621208fef204" path="/var/lib/kubelet/pods/30197fd5-0e3c-4c15-9536-621208fef204/volumes" Jul 24 23:31:35.438482 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.438437 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d"] Jul 24 23:31:35.439032 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.438950 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" Jul 24 23:31:35.439032 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.438970 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" Jul 24 23:31:35.439032 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.438985 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" Jul 24 23:31:35.439032 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.438995 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" Jul 24 23:31:35.439032 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.439033 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" Jul 24 23:31:35.439289 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.439042 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6feb23-9fd3-446a-a9cc-df1961363a94" containerName="istio-proxy" Jul 24 23:31:35.439289 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.439121 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" Jul 24 23:31:35.439289 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.439131 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30197fd5-0e3c-4c15-9536-621208fef204" containerName="istio-proxy" Jul 24 23:31:35.443777 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.443758 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.446433 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.446412 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6f3ce2-openshift-default-dockercfg-59tbp\"" Jul 24 23:31:35.453492 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.453466 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d"] Jul 24 23:31:35.577873 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.577838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-credential-socket\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.577873 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.577873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-podinfo\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.577901 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j5j\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-kube-api-access-k8j5j\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.577955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-certs\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.577982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-data\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.578017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-socket\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.578035 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-token\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.578069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9138431-8799-4d72-bf3f-a50061cb54c1-istiod-ca-cert\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.578135 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.578104 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-envoy\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679102 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j5j\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-kube-api-access-k8j5j\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679102 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-certs\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679367 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-data\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679367 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-socket\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679367 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-token\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679367 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9138431-8799-4d72-bf3f-a50061cb54c1-istiod-ca-cert\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679580 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-envoy\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679580 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-credential-socket\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679580 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-podinfo\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679580 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-certs\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679788 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-socket\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679843 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-data\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.679889 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.679870 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-credential-socket\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.680021 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.680003 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9138431-8799-4d72-bf3f-a50061cb54c1-istiod-ca-cert\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.681786 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.681767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-envoy\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.682051 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.682035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-podinfo\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.687755 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.687732 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-token\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.688038 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.688017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j5j\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-kube-api-access-k8j5j\") pod \"e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.754860 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.754772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:35.893868 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.893838 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d"] Jul 24 23:31:35.896288 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:31:35.896259 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9138431_8799_4d72_bf3f_a50061cb54c1.slice/crio-3fb37da7e5d9105af871dd9d6cb280fdbc48a5537c44859ce12976c3c15a7524 WatchSource:0}: Error finding container 3fb37da7e5d9105af871dd9d6cb280fdbc48a5537c44859ce12976c3c15a7524: Status 404 returned error can't find the container with id 3fb37da7e5d9105af871dd9d6cb280fdbc48a5537c44859ce12976c3c15a7524 Jul 24 23:31:35.898335 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.898307 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:35.898397 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.898371 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:35.898434 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:35.898401 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:36.352045 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:36.352002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" event={"ID":"e9138431-8799-4d72-bf3f-a50061cb54c1","Type":"ContainerStarted","Data":"9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e"} Jul 24 23:31:36.352203 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:36.352056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" event={"ID":"e9138431-8799-4d72-bf3f-a50061cb54c1","Type":"ContainerStarted","Data":"3fb37da7e5d9105af871dd9d6cb280fdbc48a5537c44859ce12976c3c15a7524"} Jul 24 23:31:36.371860 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:36.371799 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podStartSLOduration=1.371769249 podStartE2EDuration="1.371769249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:31:36.36974124 +0000 UTC m=+2355.644710340" watchObservedRunningTime="2026-07-24 23:31:36.371769249 +0000 UTC m=+2355.646738342" Jul 24 23:31:36.755106 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:36.755018 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:36.760075 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:36.760050 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:37.356876 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:37.356847 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:37.357890 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:37.357869 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:39.174500 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.174458 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs"] Jul 24 23:31:39.177666 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.177650 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.180879 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.180862 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-6f3ce2-dockercfg-fqjcr\"" Jul 24 23:31:39.181378 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.181365 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-6f3ce2\"" Jul 24 23:31:39.198599 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.198574 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs"] Jul 24 23:31:39.315031 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.314995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/787f9e86-38c1-432e-9f5a-99ff55677c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.315203 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.315069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln78z\" (UniqueName: \"kubernetes.io/projected/787f9e86-38c1-432e-9f5a-99ff55677c50-kube-api-access-ln78z\") pod \"payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.415697 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.415640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/787f9e86-38c1-432e-9f5a-99ff55677c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.415878 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.415736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln78z\" (UniqueName: \"kubernetes.io/projected/787f9e86-38c1-432e-9f5a-99ff55677c50-kube-api-access-ln78z\") pod \"payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.416423 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.416396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/787f9e86-38c1-432e-9f5a-99ff55677c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.427351 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.427279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln78z\" (UniqueName: \"kubernetes.io/projected/787f9e86-38c1-432e-9f5a-99ff55677c50-kube-api-access-ln78z\") pod \"payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.488065 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.488027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:39.679092 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:39.679005 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs"] Jul 24 23:31:39.681188 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:31:39.681164 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787f9e86_38c1_432e_9f5a_99ff55677c50.slice/crio-5edfbe9d5f09e882b91dfcf62c18bd233bb590ad3f880b8109f02a6d7c207959 WatchSource:0}: Error finding container 5edfbe9d5f09e882b91dfcf62c18bd233bb590ad3f880b8109f02a6d7c207959: Status 404 returned error can't find the container with id 5edfbe9d5f09e882b91dfcf62c18bd233bb590ad3f880b8109f02a6d7c207959 Jul 24 23:31:40.372328 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:40.372288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" event={"ID":"787f9e86-38c1-432e-9f5a-99ff55677c50","Type":"ContainerStarted","Data":"b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f"} Jul 24 23:31:40.372732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:40.372344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" event={"ID":"787f9e86-38c1-432e-9f5a-99ff55677c50","Type":"ContainerStarted","Data":"5edfbe9d5f09e882b91dfcf62c18bd233bb590ad3f880b8109f02a6d7c207959"} Jul 24 23:31:40.372732 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:40.372364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:40.390396 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:40.390340 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" podStartSLOduration=1.390320244 podStartE2EDuration="1.390320244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:31:40.388407962 +0000 UTC m=+2359.663377059" watchObservedRunningTime="2026-07-24 23:31:40.390320244 +0000 UTC m=+2359.665289376" Jul 24 23:31:44.318741 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.318704 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:31:44.319233 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.318779 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 23:31:44.394735 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.394703 2562 generic.go:356] "Generic (PLEG): container finished" podID="78faec82-39d5-478b-bb74-31f2f191e523" containerID="5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407" exitCode=0 Jul 24 23:31:44.394897 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.394775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" event={"ID":"78faec82-39d5-478b-bb74-31f2f191e523","Type":"ContainerDied","Data":"5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407"} Jul 24 23:31:44.395217 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.395199 2562 scope.go:117] "RemoveContainer" containerID="5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407" Jul 24 23:31:44.396194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.396169 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:44.396251 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.396236 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:44.396288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:44.396266 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:45.400514 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:45.400463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" event={"ID":"78faec82-39d5-478b-bb74-31f2f191e523","Type":"ContainerStarted","Data":"eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090"} Jul 24 23:31:45.533699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:45.529241 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:45.533699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:45.529366 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:31:45.533699 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:45.529409 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 23:31:46.528897 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:46.528853 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:31:46.529508 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:46.528939 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 23:31:47.529158 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:47.529120 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:31:47.529761 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:47.529730 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 23:31:48.529120 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:48.529077 2562 patch_prober.go:28] interesting pod/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:31:48.529339 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:48.529147 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 24 23:31:49.533716 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:49.533618 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:49.534111 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:49.533917 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:49.534591 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:49.534576 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:31:51.380579 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:51.380544 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:31:52.359070 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.359031 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:31:52.359306 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.359086 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 23:31:52.434934 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.434903 2562 generic.go:356] "Generic (PLEG): container finished" podID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerID="9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e" exitCode=0 Jul 24 23:31:52.435287 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.434979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" event={"ID":"e9138431-8799-4d72-bf3f-a50061cb54c1","Type":"ContainerDied","Data":"9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e"} Jul 24 23:31:52.435399 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.435384 2562 scope.go:117] "RemoveContainer" containerID="9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e" Jul 24 23:31:52.436432 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.436402 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:52.436499 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.436484 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:52.436579 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:52.436521 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:31:53.444118 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:53.444076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" event={"ID":"e9138431-8799-4d72-bf3f-a50061cb54c1","Type":"ContainerStarted","Data":"19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a"} Jul 24 23:31:53.755609 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:53.755512 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:53.755810 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:53.755788 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:31:53.755857 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:53.755834 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 23:31:54.755571 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:54.755538 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:31:54.755980 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:54.755590 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 23:31:55.756177 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:55.756142 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:31:55.756566 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:55.756195 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 23:31:56.755767 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:56.755730 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:31:56.755940 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:56.755788 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 24 23:31:57.759970 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:57.759937 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:57.760489 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:57.760291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:31:57.760905 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:31:57.760883 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:32:21.283592 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:32:21.283560 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:32:21.287288 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:32:21.287260 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:36:21.468373 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:36:21.468335 2562 scope.go:117] "RemoveContainer" containerID="f8ff38cde84cf77e4db8ed5b21355470faf959944696cea9a46023032b71c63f" Jul 24 23:37:21.311163 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:37:21.311134 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:37:21.315984 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:37:21.315965 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:38:11.526797 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.526758 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp"] Jul 24 23:38:11.530111 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.530091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.532928 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.532907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0aa0e6\"" Jul 24 23:38:11.533073 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.533057 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0aa0e6-dockercfg-vt6hp\"" Jul 24 23:38:11.542254 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.542227 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp"] Jul 24 23:38:11.547249 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.547225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87faf705-25b0-4afe-a412-9ce9958dd0af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.547347 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.547259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9b7\" (UniqueName: \"kubernetes.io/projected/87faf705-25b0-4afe-a412-9ce9958dd0af-kube-api-access-9t9b7\") pod \"payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.648074 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.648038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87faf705-25b0-4afe-a412-9ce9958dd0af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.648241 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.648087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9b7\" (UniqueName: \"kubernetes.io/projected/87faf705-25b0-4afe-a412-9ce9958dd0af-kube-api-access-9t9b7\") pod \"payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.648730 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.648707 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87faf705-25b0-4afe-a412-9ce9958dd0af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.656797 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.656773 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9b7\" (UniqueName: \"kubernetes.io/projected/87faf705-25b0-4afe-a412-9ce9958dd0af-kube-api-access-9t9b7\") pod \"payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.841049 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.841014 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:11.977739 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:11.977714 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp"] Jul 24 23:38:11.980147 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:38:11.980114 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87faf705_25b0_4afe_a412_9ce9958dd0af.slice/crio-f8e8c1b5a308f0d392c46340d6fbdc011c20ce429c963871f8985c35dcefff1f WatchSource:0}: Error finding container f8e8c1b5a308f0d392c46340d6fbdc011c20ce429c963871f8985c35dcefff1f: Status 404 returned error can't find the container with id f8e8c1b5a308f0d392c46340d6fbdc011c20ce429c963871f8985c35dcefff1f Jul 24 23:38:12.098991 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:12.098896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" event={"ID":"87faf705-25b0-4afe-a412-9ce9958dd0af","Type":"ContainerStarted","Data":"ba2cb2934f79685107f7bb3f07f95e6b001f53d884b95df4dfda206242241b1c"} Jul 24 23:38:12.098991 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:12.098955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:12.098991 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:12.098976 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" event={"ID":"87faf705-25b0-4afe-a412-9ce9958dd0af","Type":"ContainerStarted","Data":"f8e8c1b5a308f0d392c46340d6fbdc011c20ce429c963871f8985c35dcefff1f"} Jul 24 23:38:12.144162 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:12.144114 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" podStartSLOduration=1.144098098 podStartE2EDuration="1.144098098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:12.142111413 +0000 UTC m=+2751.417080508" watchObservedRunningTime="2026-07-24 23:38:12.144098098 +0000 UTC m=+2751.419067217" Jul 24 23:38:23.105394 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:23.105361 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:38:26.136389 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.136353 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb"] Jul 24 23:38:26.141105 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.141084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.143812 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.143790 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3fb847-openshift-default-dockercfg-hknwz\"" Jul 24 23:38:26.149978 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.149949 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb"] Jul 24 23:38:26.278310 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-envoy\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278310 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkh2\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-kube-api-access-6gkh2\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278558 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istiod-ca-cert\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278558 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-data\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278558 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-certs\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278558 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-podinfo\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278805 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-credential-socket\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278805 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-token\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.278805 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.278756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-socket\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380167 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-credential-socket\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-token\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-socket\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-envoy\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkh2\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-kube-api-access-6gkh2\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istiod-ca-cert\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380329 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380316 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-data\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380361 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-certs\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380378 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-podinfo\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380695 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-credential-socket\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380749 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-data\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-certs\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.380986 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.380970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-socket\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.381408 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.381384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istiod-ca-cert\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.382976 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.382957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-podinfo\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.383406 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.383384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-envoy\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.389260 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.389200 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-token\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.389882 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.389860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkh2\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-kube-api-access-6gkh2\") pod \"e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.453850 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.453816 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:26.590485 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.590455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb"] Jul 24 23:38:26.592516 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:38:26.592490 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5d3251_21d6_4d00_9309_3907a1cadc0f.slice/crio-af152f0b3deffecd66bc7b2ea27ad2ab32b7a0523585776dfd1631c95468f32e WatchSource:0}: Error finding container af152f0b3deffecd66bc7b2ea27ad2ab32b7a0523585776dfd1631c95468f32e: Status 404 returned error can't find the container with id af152f0b3deffecd66bc7b2ea27ad2ab32b7a0523585776dfd1631c95468f32e Jul 24 23:38:26.595218 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.595189 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:38:26.595291 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.595251 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:38:26.595291 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:26.595278 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:38:27.168915 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:27.168876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" event={"ID":"ff5d3251-21d6-4d00-9309-3907a1cadc0f","Type":"ContainerStarted","Data":"913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a"} Jul 24 23:38:27.169297 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:27.168940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" event={"ID":"ff5d3251-21d6-4d00-9309-3907a1cadc0f","Type":"ContainerStarted","Data":"af152f0b3deffecd66bc7b2ea27ad2ab32b7a0523585776dfd1631c95468f32e"} Jul 24 23:38:27.189109 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:27.189056 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" podStartSLOduration=1.189039392 podStartE2EDuration="1.189039392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:27.188162322 +0000 UTC m=+2766.463131419" watchObservedRunningTime="2026-07-24 23:38:27.189039392 +0000 UTC m=+2766.464008486" Jul 24 23:38:27.454073 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:27.453990 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:27.458952 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:27.458925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:28.173817 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:28.173790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:28.174660 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:28.174641 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:30.121159 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.121129 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2"] Jul 24 23:38:30.124282 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.124264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.126873 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.126850 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3fb847\"" Jul 24 23:38:30.127441 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.127421 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3fb847-dockercfg-gvd8l\"" Jul 24 23:38:30.136356 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.136331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2"] Jul 24 23:38:30.224212 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.224179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57lj\" (UniqueName: \"kubernetes.io/projected/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-kube-api-access-h57lj\") pod \"payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.224386 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.224220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.325123 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.325083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57lj\" (UniqueName: \"kubernetes.io/projected/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-kube-api-access-h57lj\") pod \"payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.325123 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.325128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.325905 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.325881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.334120 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.334091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57lj\" (UniqueName: \"kubernetes.io/projected/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-kube-api-access-h57lj\") pod \"payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.435334 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.435245 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:30.574643 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:30.574618 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2"] Jul 24 23:38:30.576988 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:38:30.576956 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275642ef_82c7_43fc_9ac6_fc6f70a9dab4.slice/crio-40554b8c0b4de92fa12fcb8d1da195e7dadd31bb8f871d1867e91a7adfdaebb4 WatchSource:0}: Error finding container 40554b8c0b4de92fa12fcb8d1da195e7dadd31bb8f871d1867e91a7adfdaebb4: Status 404 returned error can't find the container with id 40554b8c0b4de92fa12fcb8d1da195e7dadd31bb8f871d1867e91a7adfdaebb4 Jul 24 23:38:31.193028 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:31.192994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" event={"ID":"275642ef-82c7-43fc-9ac6-fc6f70a9dab4","Type":"ContainerStarted","Data":"eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc"} Jul 24 23:38:31.193412 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:31.193046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" event={"ID":"275642ef-82c7-43fc-9ac6-fc6f70a9dab4","Type":"ContainerStarted","Data":"40554b8c0b4de92fa12fcb8d1da195e7dadd31bb8f871d1867e91a7adfdaebb4"} Jul 24 23:38:31.193412 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:31.193073 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:31.208670 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:31.208622 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" podStartSLOduration=1.208603362 podStartE2EDuration="1.208603362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:31.208080877 +0000 UTC m=+2770.483049975" watchObservedRunningTime="2026-07-24 23:38:31.208603362 +0000 UTC m=+2770.483572458" Jul 24 23:38:42.199803 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:42.199773 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:38:48.285627 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:48.285590 2562 generic.go:356] "Generic (PLEG): container finished" podID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerID="913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a" exitCode=0 Jul 24 23:38:48.286061 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:48.285655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" event={"ID":"ff5d3251-21d6-4d00-9309-3907a1cadc0f","Type":"ContainerDied","Data":"913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a"} Jul 24 23:38:48.286142 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:48.286126 2562 scope.go:117] "RemoveContainer" containerID="913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a" Jul 24 23:38:48.287189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:48.287164 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:38:48.287253 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:48.287230 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:38:48.287290 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:48.287261 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:38:49.291833 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:49.291796 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" event={"ID":"ff5d3251-21d6-4d00-9309-3907a1cadc0f","Type":"ContainerStarted","Data":"7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1"} Jul 24 23:38:49.454733 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:49.454686 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:49.454940 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:49.454917 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 24 23:38:49.455002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:49.454978 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 24 23:38:50.454722 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:50.454653 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 24 23:38:50.455200 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:50.454740 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 24 23:38:50.818463 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:50.818366 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ndrmz_d4090740-da29-4dad-b5b7-8a587436717d/payload-processing/0.log" Jul 24 23:38:51.454604 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:51.454562 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 24 23:38:51.454827 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:51.454635 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 24 23:38:52.454786 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:52.454746 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 24 23:38:52.454986 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:52.454832 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 24 23:38:53.458381 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:53.458352 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:53.458768 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:53.458655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:38:53.459292 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:38:53.459275 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:39:37.873530 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:37.873494 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ndrmz_d4090740-da29-4dad-b5b7-8a587436717d/payload-processing/0.log" Jul 24 23:39:38.559782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.559746 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs"] Jul 24 23:39:38.563498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.563474 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.566111 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.566086 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-4bb47e-openshift-default-dockercfg-jt664\"" Jul 24 23:39:38.575968 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.575942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs"] Jul 24 23:39:38.635602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-token\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxz8\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-kube-api-access-7kxz8\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-data\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-podinfo\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-envoy\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-socket\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.635936 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-credential-socket\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.636081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.635937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-certs\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737348 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxz8\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-kube-api-access-7kxz8\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-data\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-podinfo\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-envoy\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-socket\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737851 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-credential-socket\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737851 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-certs\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.737851 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-token\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.738015 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737873 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-data\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.738015 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.737973 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-socket\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.738130 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.738099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-certs\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.738201 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.738186 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-credential-socket\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.738311 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.738292 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.740028 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.740006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-envoy\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.740312 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.740293 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-podinfo\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.746037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.746016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-token\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.746037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.746028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxz8\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-kube-api-access-7kxz8\") pod \"e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:38.875121 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:38.875078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:39.016325 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.016291 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs"] Jul 24 23:39:39.017825 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:39:39.017799 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbd8504_5cfd_4adb_8479_a73832b79f0f.slice/crio-9ba5921c9e78097367edf400872a0f699a4c1126237b751715d6c435119abcc8 WatchSource:0}: Error finding container 9ba5921c9e78097367edf400872a0f699a4c1126237b751715d6c435119abcc8: Status 404 returned error can't find the container with id 9ba5921c9e78097367edf400872a0f699a4c1126237b751715d6c435119abcc8 Jul 24 23:39:39.020081 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.020051 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:39:39.020152 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.020117 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:39:39.020152 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.020143 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:39:39.529362 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.529324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" event={"ID":"cfbd8504-5cfd-4adb-8479-a73832b79f0f","Type":"ContainerStarted","Data":"9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762"} Jul 24 23:39:39.529537 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.529380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" event={"ID":"cfbd8504-5cfd-4adb-8479-a73832b79f0f","Type":"ContainerStarted","Data":"9ba5921c9e78097367edf400872a0f699a4c1126237b751715d6c435119abcc8"} Jul 24 23:39:39.552363 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.552310 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" podStartSLOduration=1.552292977 podStartE2EDuration="1.552292977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:39:39.55078513 +0000 UTC m=+2838.825754238" watchObservedRunningTime="2026-07-24 23:39:39.552292977 +0000 UTC m=+2838.827262071" Jul 24 23:39:39.875175 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:39.875132 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:40.881296 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:40.881262 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:41.539217 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:41.539183 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:41.540038 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:41.540018 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:42.238987 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.238952 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7"] Jul 24 23:39:42.242324 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.242303 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.247755 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.247732 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4bb47e\"" Jul 24 23:39:42.261993 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.261970 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7"] Jul 24 23:39:42.272268 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.272244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twtwb\" (UniqueName: \"kubernetes.io/projected/d02db8e2-f411-4c01-a230-fbb72ff0d209-kube-api-access-twtwb\") pod \"payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.272375 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.272296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02db8e2-f411-4c01-a230-fbb72ff0d209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.373213 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.373163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twtwb\" (UniqueName: \"kubernetes.io/projected/d02db8e2-f411-4c01-a230-fbb72ff0d209-kube-api-access-twtwb\") pod \"payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.374100 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.373905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02db8e2-f411-4c01-a230-fbb72ff0d209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.375637 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.375572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02db8e2-f411-4c01-a230-fbb72ff0d209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.383810 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.383738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twtwb\" (UniqueName: \"kubernetes.io/projected/d02db8e2-f411-4c01-a230-fbb72ff0d209-kube-api-access-twtwb\") pod \"payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.552752 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.552648 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:42.695358 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:42.694085 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7"] Jul 24 23:39:42.700394 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:39:42.700364 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02db8e2_f411_4c01_a230_fbb72ff0d209.slice/crio-f6e1e3d4e6ff0cce0ef200da6dee330c2ef86f8f33adaa259f7cfb7235ccb33c WatchSource:0}: Error finding container f6e1e3d4e6ff0cce0ef200da6dee330c2ef86f8f33adaa259f7cfb7235ccb33c: Status 404 returned error can't find the container with id f6e1e3d4e6ff0cce0ef200da6dee330c2ef86f8f33adaa259f7cfb7235ccb33c Jul 24 23:39:43.549327 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:43.549291 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" event={"ID":"d02db8e2-f411-4c01-a230-fbb72ff0d209","Type":"ContainerStarted","Data":"a52eaa6c213f12e01f42f1b64e31c1174a5a6052fbe721c713f94e524ca83f21"} Jul 24 23:39:43.549841 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:43.549349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" event={"ID":"d02db8e2-f411-4c01-a230-fbb72ff0d209","Type":"ContainerStarted","Data":"f6e1e3d4e6ff0cce0ef200da6dee330c2ef86f8f33adaa259f7cfb7235ccb33c"} Jul 24 23:39:43.549841 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:43.549523 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:43.566964 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:43.566918 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" podStartSLOduration=1.566903191 podStartE2EDuration="1.566903191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:39:43.564457482 +0000 UTC m=+2842.839426576" watchObservedRunningTime="2026-07-24 23:39:43.566903191 +0000 UTC m=+2842.841872286" Jul 24 23:39:54.556429 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:54.556395 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:39:55.610548 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:55.610466 2562 generic.go:356] "Generic (PLEG): container finished" podID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerID="9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762" exitCode=0 Jul 24 23:39:55.610970 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:55.610547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" event={"ID":"cfbd8504-5cfd-4adb-8479-a73832b79f0f","Type":"ContainerDied","Data":"9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762"} Jul 24 23:39:55.610970 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:55.610960 2562 scope.go:117] "RemoveContainer" containerID="9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762" Jul 24 23:39:55.612008 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:55.611981 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:39:55.612125 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:55.612084 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:39:55.612189 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:55.612132 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 23:39:56.539168 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:56.539115 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:56.616533 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:56.616498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" event={"ID":"cfbd8504-5cfd-4adb-8479-a73832b79f0f","Type":"ContainerStarted","Data":"c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c"} Jul 24 23:39:56.876019 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:56.875972 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:39:56.877358 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:56.877328 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 24 23:39:56.877478 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:56.877394 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 24 23:39:57.876165 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:57.876130 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 24 23:39:57.876540 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:57.876196 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 24 23:39:58.876202 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:58.876167 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 24 23:39:58.876602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:58.876220 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 24 23:39:59.875781 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:59.875741 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 24 23:39:59.876052 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:39:59.875801 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 24 23:40:00.879921 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:00.879890 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:40:00.880305 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:00.880178 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:40:00.880836 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:00.880814 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:40:07.466858 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.466813 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7"] Jul 24 23:40:07.467306 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.467055 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" podUID="d02db8e2-f411-4c01-a230-fbb72ff0d209" containerName="payload-pre-processing" containerID="cri-o://a52eaa6c213f12e01f42f1b64e31c1174a5a6052fbe721c713f94e524ca83f21" gracePeriod=30 Jul 24 23:40:07.695216 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.695183 2562 generic.go:356] "Generic (PLEG): container finished" podID="d02db8e2-f411-4c01-a230-fbb72ff0d209" containerID="a52eaa6c213f12e01f42f1b64e31c1174a5a6052fbe721c713f94e524ca83f21" exitCode=0 Jul 24 23:40:07.695216 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.695222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" event={"ID":"d02db8e2-f411-4c01-a230-fbb72ff0d209","Type":"ContainerDied","Data":"a52eaa6c213f12e01f42f1b64e31c1174a5a6052fbe721c713f94e524ca83f21"} Jul 24 23:40:07.741064 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.741043 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:40:07.920494 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.920458 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02db8e2-f411-4c01-a230-fbb72ff0d209-plugins-config-volume\") pod \"d02db8e2-f411-4c01-a230-fbb72ff0d209\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " Jul 24 23:40:07.920758 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.920535 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twtwb\" (UniqueName: \"kubernetes.io/projected/d02db8e2-f411-4c01-a230-fbb72ff0d209-kube-api-access-twtwb\") pod \"d02db8e2-f411-4c01-a230-fbb72ff0d209\" (UID: \"d02db8e2-f411-4c01-a230-fbb72ff0d209\") " Jul 24 23:40:07.920867 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.920842 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02db8e2-f411-4c01-a230-fbb72ff0d209-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d02db8e2-f411-4c01-a230-fbb72ff0d209" (UID: "d02db8e2-f411-4c01-a230-fbb72ff0d209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:40:07.922527 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:07.922503 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02db8e2-f411-4c01-a230-fbb72ff0d209-kube-api-access-twtwb" (OuterVolumeSpecName: "kube-api-access-twtwb") pod "d02db8e2-f411-4c01-a230-fbb72ff0d209" (UID: "d02db8e2-f411-4c01-a230-fbb72ff0d209"). InnerVolumeSpecName "kube-api-access-twtwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:40:08.021459 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.021376 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02db8e2-f411-4c01-a230-fbb72ff0d209-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:08.021459 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.021405 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twtwb\" (UniqueName: \"kubernetes.io/projected/d02db8e2-f411-4c01-a230-fbb72ff0d209-kube-api-access-twtwb\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:08.701144 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.701112 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" Jul 24 23:40:08.701144 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.701128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7" event={"ID":"d02db8e2-f411-4c01-a230-fbb72ff0d209","Type":"ContainerDied","Data":"f6e1e3d4e6ff0cce0ef200da6dee330c2ef86f8f33adaa259f7cfb7235ccb33c"} Jul 24 23:40:08.701628 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.701195 2562 scope.go:117] "RemoveContainer" containerID="a52eaa6c213f12e01f42f1b64e31c1174a5a6052fbe721c713f94e524ca83f21" Jul 24 23:40:08.727929 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.727901 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7"] Jul 24 23:40:08.732011 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:08.731984 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4bb47e-68d9984b48-k7fg7"] Jul 24 23:40:09.300335 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:09.300301 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02db8e2-f411-4c01-a230-fbb72ff0d209" path="/var/lib/kubelet/pods/d02db8e2-f411-4c01-a230-fbb72ff0d209/volumes" Jul 24 23:40:18.266305 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:18.266267 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs"] Jul 24 23:40:18.266849 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:18.266538 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" containerID="cri-o://c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c" gracePeriod=30 Jul 24 23:40:23.526853 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.526825 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551468 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-certs\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-envoy\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551591 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-token\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kxz8\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-kube-api-access-7kxz8\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551660 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-podinfo\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-data\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551760 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-socket\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551791 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istiod-ca-cert\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.552407 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.551828 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-credential-socket\") pod \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\" (UID: \"cfbd8504-5cfd-4adb-8479-a73832b79f0f\") " Jul 24 23:40:23.553011 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.552933 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:40:23.553159 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.553057 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:40:23.553159 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.553124 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:40:23.553290 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.553272 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:40:23.553485 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.553454 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:40:23.555260 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.555230 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:40:23.555260 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.555243 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:40:23.555260 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.555252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:40:23.555646 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.555614 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-kube-api-access-7kxz8" (OuterVolumeSpecName: "kube-api-access-7kxz8") pod "cfbd8504-5cfd-4adb-8479-a73832b79f0f" (UID: "cfbd8504-5cfd-4adb-8479-a73832b79f0f"). InnerVolumeSpecName "kube-api-access-7kxz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:40:23.652826 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652797 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.652826 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652823 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.652826 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652832 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.653077 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652842 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxz8\" (UniqueName: \"kubernetes.io/projected/cfbd8504-5cfd-4adb-8479-a73832b79f0f-kube-api-access-7kxz8\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.653077 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652852 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.653077 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652860 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.653077 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652868 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.653077 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652876 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbd8504-5cfd-4adb-8479-a73832b79f0f-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.653077 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.652885 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbd8504-5cfd-4adb-8479-a73832b79f0f-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:23.772613 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.772578 2562 generic.go:356] "Generic (PLEG): container finished" podID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerID="c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c" exitCode=0 Jul 24 23:40:23.772821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.772638 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" Jul 24 23:40:23.772821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.772655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" event={"ID":"cfbd8504-5cfd-4adb-8479-a73832b79f0f","Type":"ContainerDied","Data":"c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c"} Jul 24 23:40:23.772821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.772746 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs" event={"ID":"cfbd8504-5cfd-4adb-8479-a73832b79f0f","Type":"ContainerDied","Data":"9ba5921c9e78097367edf400872a0f699a4c1126237b751715d6c435119abcc8"} Jul 24 23:40:23.772821 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.772779 2562 scope.go:117] "RemoveContainer" containerID="c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c" Jul 24 23:40:23.782003 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.781986 2562 scope.go:117] "RemoveContainer" containerID="9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762" Jul 24 23:40:23.789892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.789875 2562 scope.go:117] "RemoveContainer" containerID="c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c" Jul 24 23:40:23.790136 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:40:23.790117 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c\": container with ID starting with c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c not found: ID does not exist" containerID="c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c" Jul 24 23:40:23.790217 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.790149 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c"} err="failed to get container status \"c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c\": rpc error: code = NotFound desc = could not find container \"c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c\": container with ID starting with c05141539a2e5754dd7a7bbe29889ad4e3e625fc932a030e4dd6ecbe053a9a7c not found: ID does not exist" Jul 24 23:40:23.790217 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.790176 2562 scope.go:117] "RemoveContainer" containerID="9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762" Jul 24 23:40:23.790418 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:40:23.790402 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762\": container with ID starting with 9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762 not found: ID does not exist" containerID="9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762" Jul 24 23:40:23.790474 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.790425 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762"} err="failed to get container status \"9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762\": rpc error: code = NotFound desc = could not find container \"9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762\": container with ID starting with 9c7906b00b3d8b8595edba0d30f379a6283df8efb6a0fffb21b870b46a71e762 not found: ID does not exist" Jul 24 23:40:23.794498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.794475 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs"] Jul 24 23:40:23.798670 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:23.798648 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4bb47e-openshift-default-6f95b8f4d8-xqkqs"] Jul 24 23:40:25.299584 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:25.299538 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" path="/var/lib/kubelet/pods/cfbd8504-5cfd-4adb-8479-a73832b79f0f/volumes" Jul 24 23:40:37.563976 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.563939 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp"] Jul 24 23:40:37.564459 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.564227 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" podUID="87faf705-25b0-4afe-a412-9ce9958dd0af" containerName="payload-pre-processing" containerID="cri-o://ba2cb2934f79685107f7bb3f07f95e6b001f53d884b95df4dfda206242241b1c" gracePeriod=30 Jul 24 23:40:37.836978 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.836944 2562 generic.go:356] "Generic (PLEG): container finished" podID="87faf705-25b0-4afe-a412-9ce9958dd0af" containerID="ba2cb2934f79685107f7bb3f07f95e6b001f53d884b95df4dfda206242241b1c" exitCode=0 Jul 24 23:40:37.837108 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.837013 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" event={"ID":"87faf705-25b0-4afe-a412-9ce9958dd0af","Type":"ContainerDied","Data":"ba2cb2934f79685107f7bb3f07f95e6b001f53d884b95df4dfda206242241b1c"} Jul 24 23:40:37.837108 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.837033 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:40:37.837108 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.837062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" event={"ID":"87faf705-25b0-4afe-a412-9ce9958dd0af","Type":"ContainerDied","Data":"f8e8c1b5a308f0d392c46340d6fbdc011c20ce429c963871f8985c35dcefff1f"} Jul 24 23:40:37.837108 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.837079 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e8c1b5a308f0d392c46340d6fbdc011c20ce429c963871f8985c35dcefff1f" Jul 24 23:40:37.882282 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.882249 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t9b7\" (UniqueName: \"kubernetes.io/projected/87faf705-25b0-4afe-a412-9ce9958dd0af-kube-api-access-9t9b7\") pod \"87faf705-25b0-4afe-a412-9ce9958dd0af\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " Jul 24 23:40:37.882440 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.882320 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87faf705-25b0-4afe-a412-9ce9958dd0af-plugins-config-volume\") pod \"87faf705-25b0-4afe-a412-9ce9958dd0af\" (UID: \"87faf705-25b0-4afe-a412-9ce9958dd0af\") " Jul 24 23:40:37.882791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.882755 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87faf705-25b0-4afe-a412-9ce9958dd0af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87faf705-25b0-4afe-a412-9ce9958dd0af" (UID: "87faf705-25b0-4afe-a412-9ce9958dd0af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:40:37.884235 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.884215 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87faf705-25b0-4afe-a412-9ce9958dd0af-kube-api-access-9t9b7" (OuterVolumeSpecName: "kube-api-access-9t9b7") pod "87faf705-25b0-4afe-a412-9ce9958dd0af" (UID: "87faf705-25b0-4afe-a412-9ce9958dd0af"). InnerVolumeSpecName "kube-api-access-9t9b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:40:37.983225 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.983193 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87faf705-25b0-4afe-a412-9ce9958dd0af-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:37.983225 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:37.983219 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9b7\" (UniqueName: \"kubernetes.io/projected/87faf705-25b0-4afe-a412-9ce9958dd0af-kube-api-access-9t9b7\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:40:38.841522 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:38.841491 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp" Jul 24 23:40:38.861384 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:38.861355 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp"] Jul 24 23:40:38.867197 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:38.867165 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0aa0e6-8499766b56-4w2jp"] Jul 24 23:40:39.300144 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:40:39.300064 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87faf705-25b0-4afe-a412-9ce9958dd0af" path="/var/lib/kubelet/pods/87faf705-25b0-4afe-a412-9ce9958dd0af/volumes" Jul 24 23:41:46.664069 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:46.663983 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2"] Jul 24 23:41:46.664597 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:46.664240 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" podUID="275642ef-82c7-43fc-9ac6-fc6f70a9dab4" containerName="payload-pre-processing" containerID="cri-o://eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc" gracePeriod=30 Jul 24 23:41:46.927161 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:46.927100 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:41:47.099371 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.099337 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h57lj\" (UniqueName: \"kubernetes.io/projected/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-kube-api-access-h57lj\") pod \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " Jul 24 23:41:47.099539 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.099447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-plugins-config-volume\") pod \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\" (UID: \"275642ef-82c7-43fc-9ac6-fc6f70a9dab4\") " Jul 24 23:41:47.099877 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.099843 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "275642ef-82c7-43fc-9ac6-fc6f70a9dab4" (UID: "275642ef-82c7-43fc-9ac6-fc6f70a9dab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:41:47.101418 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.101387 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-kube-api-access-h57lj" (OuterVolumeSpecName: "kube-api-access-h57lj") pod "275642ef-82c7-43fc-9ac6-fc6f70a9dab4" (UID: "275642ef-82c7-43fc-9ac6-fc6f70a9dab4"). InnerVolumeSpecName "kube-api-access-h57lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:41:47.148503 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.148468 2562 generic.go:356] "Generic (PLEG): container finished" podID="275642ef-82c7-43fc-9ac6-fc6f70a9dab4" containerID="eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc" exitCode=0 Jul 24 23:41:47.148702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.148516 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" Jul 24 23:41:47.148702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.148562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" event={"ID":"275642ef-82c7-43fc-9ac6-fc6f70a9dab4","Type":"ContainerDied","Data":"eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc"} Jul 24 23:41:47.148702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.148617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2" event={"ID":"275642ef-82c7-43fc-9ac6-fc6f70a9dab4","Type":"ContainerDied","Data":"40554b8c0b4de92fa12fcb8d1da195e7dadd31bb8f871d1867e91a7adfdaebb4"} Jul 24 23:41:47.148702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.148641 2562 scope.go:117] "RemoveContainer" containerID="eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc" Jul 24 23:41:47.158155 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.158139 2562 scope.go:117] "RemoveContainer" containerID="eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc" Jul 24 23:41:47.158410 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:41:47.158392 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc\": container with ID starting with eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc not found: ID does not exist" containerID="eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc" Jul 24 23:41:47.158468 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.158419 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc"} err="failed to get container status \"eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc\": rpc error: code = NotFound desc = could not find container \"eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc\": container with ID starting with eb8c5a37e879a55040985cba6b311a0927976f11b63e2ec5c2cc96136f8420bc not found: ID does not exist" Jul 24 23:41:47.170657 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.170630 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2"] Jul 24 23:41:47.174942 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.174920 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3fb847-6bb4889c76-f2gd2"] Jul 24 23:41:47.200912 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.200840 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:41:47.200912 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.200866 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h57lj\" (UniqueName: \"kubernetes.io/projected/275642ef-82c7-43fc-9ac6-fc6f70a9dab4-kube-api-access-h57lj\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:41:47.301354 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:47.301315 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="275642ef-82c7-43fc-9ac6-fc6f70a9dab4" path="/var/lib/kubelet/pods/275642ef-82c7-43fc-9ac6-fc6f70a9dab4/volumes" Jul 24 23:41:57.699838 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:57.699805 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb"] Jul 24 23:41:57.700363 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:41:57.700000 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" containerID="cri-o://7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1" gracePeriod=30 Jul 24 23:42:02.956390 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:02.956362 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:42:03.043593 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043508 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istiod-ca-cert\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043593 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043566 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-data\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043593 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043582 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-credential-socket\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043609 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gkh2\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-kube-api-access-6gkh2\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043635 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-envoy\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043650 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-certs\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043670 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-socket\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043751 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-podinfo\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.043919 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.043772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-token\") pod \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\" (UID: \"ff5d3251-21d6-4d00-9309-3907a1cadc0f\") " Jul 24 23:42:03.044197 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.044021 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:03.044197 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.044036 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:03.044197 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.044101 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:42:03.044197 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.044119 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:03.044506 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.044480 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:03.046059 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.046040 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:42:03.046145 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.046053 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:03.046237 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.046219 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:03.046291 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.046242 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-kube-api-access-6gkh2" (OuterVolumeSpecName: "kube-api-access-6gkh2") pod "ff5d3251-21d6-4d00-9309-3907a1cadc0f" (UID: "ff5d3251-21d6-4d00-9309-3907a1cadc0f"). InnerVolumeSpecName "kube-api-access-6gkh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:03.144765 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144714 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.144765 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144759 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.144765 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144773 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.145026 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144785 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkh2\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-kube-api-access-6gkh2\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.145026 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144798 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.145026 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144811 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.145026 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144822 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff5d3251-21d6-4d00-9309-3907a1cadc0f-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.145026 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144833 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.145026 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.144843 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff5d3251-21d6-4d00-9309-3907a1cadc0f-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:03.224579 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.224545 2562 generic.go:356] "Generic (PLEG): container finished" podID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerID="7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1" exitCode=0 Jul 24 23:42:03.224785 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.224595 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" Jul 24 23:42:03.224785 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.224614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" event={"ID":"ff5d3251-21d6-4d00-9309-3907a1cadc0f","Type":"ContainerDied","Data":"7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1"} Jul 24 23:42:03.224785 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.224657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb" event={"ID":"ff5d3251-21d6-4d00-9309-3907a1cadc0f","Type":"ContainerDied","Data":"af152f0b3deffecd66bc7b2ea27ad2ab32b7a0523585776dfd1631c95468f32e"} Jul 24 23:42:03.224785 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.224714 2562 scope.go:117] "RemoveContainer" containerID="7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1" Jul 24 23:42:03.235906 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.235885 2562 scope.go:117] "RemoveContainer" containerID="913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a" Jul 24 23:42:03.243722 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.243705 2562 scope.go:117] "RemoveContainer" containerID="7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1" Jul 24 23:42:03.243982 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:03.243961 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1\": container with ID starting with 7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1 not found: ID does not exist" containerID="7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1" Jul 24 23:42:03.244069 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.243989 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1"} err="failed to get container status \"7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1\": rpc error: code = NotFound desc = could not find container \"7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1\": container with ID starting with 7eef42e85564101c870a3fa8777556707b7fcdb9a3dcbae198a154c2713bf5e1 not found: ID does not exist" Jul 24 23:42:03.244069 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.244014 2562 scope.go:117] "RemoveContainer" containerID="913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a" Jul 24 23:42:03.244310 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:03.244286 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a\": container with ID starting with 913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a not found: ID does not exist" containerID="913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a" Jul 24 23:42:03.244375 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.244319 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a"} err="failed to get container status \"913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a\": rpc error: code = NotFound desc = could not find container \"913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a\": container with ID starting with 913aee18bafae892111d394387e602c2e4643008baf61a7c5c4e111259eae05a not found: ID does not exist" Jul 24 23:42:03.251239 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.251217 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb"] Jul 24 23:42:03.255181 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.255160 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3fb847-openshift-default-5f69f8b4b4-t9drb"] Jul 24 23:42:03.299758 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:03.299689 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" path="/var/lib/kubelet/pods/ff5d3251-21d6-4d00-9309-3907a1cadc0f/volumes" Jul 24 23:42:19.611666 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:19.611626 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc"] Jul 24 23:42:19.612078 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:19.611897 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" containerID="cri-o://eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090" gracePeriod=30 Jul 24 23:42:21.335914 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:21.335885 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:42:21.341864 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:21.341839 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7wkj_5dad8c34-9078-4e76-8f4c-5b1590840294/ovn-acl-logging/0.log" Jul 24 23:42:24.855059 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:24.855031 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:42:25.038591 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038506 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-certs\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038591 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038545 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78faec82-39d5-478b-bb74-31f2f191e523-istiod-ca-cert\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038594 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cfht\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-kube-api-access-6cfht\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038612 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-istio-token\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038650 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-envoy\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038695 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78faec82-39d5-478b-bb74-31f2f191e523-istio-podinfo\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-data\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-credential-socket\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-socket\") pod \"78faec82-39d5-478b-bb74-31f2f191e523\" (UID: \"78faec82-39d5-478b-bb74-31f2f191e523\") " Jul 24 23:42:25.038856 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.038845 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:25.039276 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.039060 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.039276 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.039104 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78faec82-39d5-478b-bb74-31f2f191e523-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:42:25.039385 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.039275 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:25.039438 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.039385 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:25.039814 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.039786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-data" (OuterVolumeSpecName: "istio-data") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:25.040927 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.040896 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-istio-token" (OuterVolumeSpecName: "istio-token") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:25.041027 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.040944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-kube-api-access-6cfht" (OuterVolumeSpecName: "kube-api-access-6cfht") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "kube-api-access-6cfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:25.041027 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.041012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:25.041027 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.041017 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78faec82-39d5-478b-bb74-31f2f191e523-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78faec82-39d5-478b-bb74-31f2f191e523" (UID: "78faec82-39d5-478b-bb74-31f2f191e523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:42:25.139604 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139571 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfht\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-kube-api-access-6cfht\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139604 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139600 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78faec82-39d5-478b-bb74-31f2f191e523-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139604 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139610 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139858 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139619 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78faec82-39d5-478b-bb74-31f2f191e523-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139858 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139627 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139858 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139635 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139858 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139643 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78faec82-39d5-478b-bb74-31f2f191e523-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.139858 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.139652 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78faec82-39d5-478b-bb74-31f2f191e523-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:25.325791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.325763 2562 generic.go:356] "Generic (PLEG): container finished" podID="78faec82-39d5-478b-bb74-31f2f191e523" containerID="eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090" exitCode=0 Jul 24 23:42:25.325950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.325815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" event={"ID":"78faec82-39d5-478b-bb74-31f2f191e523","Type":"ContainerDied","Data":"eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090"} Jul 24 23:42:25.325950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.325853 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" event={"ID":"78faec82-39d5-478b-bb74-31f2f191e523","Type":"ContainerDied","Data":"b0a45f58af650c09c907094ec1e617b3968d600097f79fea970b1b4df70b93f5"} Jul 24 23:42:25.325950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.325875 2562 scope.go:117] "RemoveContainer" containerID="eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090" Jul 24 23:42:25.325950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.325820 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc" Jul 24 23:42:25.334162 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.334138 2562 scope.go:117] "RemoveContainer" containerID="5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407" Jul 24 23:42:25.341893 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.341875 2562 scope.go:117] "RemoveContainer" containerID="eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090" Jul 24 23:42:25.342217 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:25.342192 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090\": container with ID starting with eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090 not found: ID does not exist" containerID="eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090" Jul 24 23:42:25.342289 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.342223 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090"} err="failed to get container status \"eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090\": rpc error: code = NotFound desc = could not find container \"eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090\": container with ID starting with eacc245b3fc31c1751c75e16652234f840ab03d87823320600e299a3c95ce090 not found: ID does not exist" Jul 24 23:42:25.342289 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.342244 2562 scope.go:117] "RemoveContainer" containerID="5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407" Jul 24 23:42:25.342471 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:25.342454 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407\": container with ID starting with 5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407 not found: ID does not exist" containerID="5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407" Jul 24 23:42:25.342516 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.342477 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407"} err="failed to get container status \"5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407\": rpc error: code = NotFound desc = could not find container \"5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407\": container with ID starting with 5ac6049305a82faf5e1983832c957fd07b215648766db997a3682dd505f85407 not found: ID does not exist" Jul 24 23:42:25.343018 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.342998 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc"] Jul 24 23:42:25.348514 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:25.348491 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fe475f-openshift-default-8774569c6-k62nc"] Jul 24 23:42:27.300756 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:27.300709 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78faec82-39d5-478b-bb74-31f2f191e523" path="/var/lib/kubelet/pods/78faec82-39d5-478b-bb74-31f2f191e523/volumes" Jul 24 23:42:30.534108 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.534077 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs"] Jul 24 23:42:30.535002 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.534312 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" podUID="787f9e86-38c1-432e-9f5a-99ff55677c50" containerName="payload-pre-processing" containerID="cri-o://b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f" gracePeriod=30 Jul 24 23:42:30.790317 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.790262 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:42:30.882500 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.882469 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln78z\" (UniqueName: \"kubernetes.io/projected/787f9e86-38c1-432e-9f5a-99ff55677c50-kube-api-access-ln78z\") pod \"787f9e86-38c1-432e-9f5a-99ff55677c50\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " Jul 24 23:42:30.882698 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.882533 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/787f9e86-38c1-432e-9f5a-99ff55677c50-plugins-config-volume\") pod \"787f9e86-38c1-432e-9f5a-99ff55677c50\" (UID: \"787f9e86-38c1-432e-9f5a-99ff55677c50\") " Jul 24 23:42:30.882904 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.882881 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787f9e86-38c1-432e-9f5a-99ff55677c50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "787f9e86-38c1-432e-9f5a-99ff55677c50" (UID: "787f9e86-38c1-432e-9f5a-99ff55677c50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:42:30.884556 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.884529 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787f9e86-38c1-432e-9f5a-99ff55677c50-kube-api-access-ln78z" (OuterVolumeSpecName: "kube-api-access-ln78z") pod "787f9e86-38c1-432e-9f5a-99ff55677c50" (UID: "787f9e86-38c1-432e-9f5a-99ff55677c50"). InnerVolumeSpecName "kube-api-access-ln78z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:30.983542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.983510 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln78z\" (UniqueName: \"kubernetes.io/projected/787f9e86-38c1-432e-9f5a-99ff55677c50-kube-api-access-ln78z\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:30.983542 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:30.983536 2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/787f9e86-38c1-432e-9f5a-99ff55677c50-plugins-config-volume\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:31.353334 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.353297 2562 generic.go:356] "Generic (PLEG): container finished" podID="787f9e86-38c1-432e-9f5a-99ff55677c50" containerID="b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f" exitCode=0 Jul 24 23:42:31.353498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.353356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" Jul 24 23:42:31.353498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.353388 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" event={"ID":"787f9e86-38c1-432e-9f5a-99ff55677c50","Type":"ContainerDied","Data":"b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f"} Jul 24 23:42:31.353498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.353437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs" event={"ID":"787f9e86-38c1-432e-9f5a-99ff55677c50","Type":"ContainerDied","Data":"5edfbe9d5f09e882b91dfcf62c18bd233bb590ad3f880b8109f02a6d7c207959"} Jul 24 23:42:31.353498 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.353458 2562 scope.go:117] "RemoveContainer" containerID="b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f" Jul 24 23:42:31.361463 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.361437 2562 scope.go:117] "RemoveContainer" containerID="b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f" Jul 24 23:42:31.361737 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:31.361718 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f\": container with ID starting with b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f not found: ID does not exist" containerID="b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f" Jul 24 23:42:31.361777 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.361746 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f"} err="failed to get container status \"b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f\": rpc error: code = NotFound desc = could not find container \"b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f\": container with ID starting with b5afdfa6673686a931354a189dfb4839defd227095c67da0f2fa5effee42626f not found: ID does not exist" Jul 24 23:42:31.369053 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.369032 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs"] Jul 24 23:42:31.373058 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:31.373040 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-6f3ce2-84d6f8b58d-ss6qs"] Jul 24 23:42:33.299454 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:33.299418 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787f9e86-38c1-432e-9f5a-99ff55677c50" path="/var/lib/kubelet/pods/787f9e86-38c1-432e-9f5a-99ff55677c50/volumes" Jul 24 23:42:41.392105 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:41.392076 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d"] Jul 24 23:42:41.392526 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:41.392264 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" containerID="cri-o://19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a" gracePeriod=30 Jul 24 23:42:42.763097 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:42.763053 2562 patch_prober.go:28] interesting pod/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 23:42:42.763462 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:42.763123 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 23:42:46.638808 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.638782 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:42:46.710267 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710202 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-data\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710267 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9138431-8799-4d72-bf3f-a50061cb54c1-istiod-ca-cert\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710272 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-socket\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710301 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-credential-socket\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710327 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-token\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710348 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8j5j\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-kube-api-access-k8j5j\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710372 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-certs\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710414 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-podinfo\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710452 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710444 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-envoy\") pod \"e9138431-8799-4d72-bf3f-a50061cb54c1\" (UID: \"e9138431-8799-4d72-bf3f-a50061cb54c1\") " Jul 24 23:42:46.710840 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710648 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:46.710840 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710723 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:46.710840 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710744 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9138431-8799-4d72-bf3f-a50061cb54c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:42:46.711008 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710983 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9138431-8799-4d72-bf3f-a50061cb54c1-istiod-ca-cert\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.711008 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.710991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:46.711099 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.711002 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-credential-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.711099 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.711036 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-certs\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.711167 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.711104 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:46.712651 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.712623 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:46.712770 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.712660 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:42:46.712993 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.712966 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-kube-api-access-k8j5j" (OuterVolumeSpecName: "kube-api-access-k8j5j") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "kube-api-access-k8j5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:42:46.712993 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.712966 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9138431-8799-4d72-bf3f-a50061cb54c1" (UID: "e9138431-8799-4d72-bf3f-a50061cb54c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:42:46.811764 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.811736 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-podinfo\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.811764 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.811758 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-envoy\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.811764 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.811767 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-data\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.811983 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.811774 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9138431-8799-4d72-bf3f-a50061cb54c1-workload-socket\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.811983 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.811784 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-istio-token\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:46.811983 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:46.811793 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8j5j\" (UniqueName: \"kubernetes.io/projected/e9138431-8799-4d72-bf3f-a50061cb54c1-kube-api-access-k8j5j\") on node \"ip-10-0-143-241.ec2.internal\" DevicePath \"\"" Jul 24 23:42:47.425383 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.425346 2562 generic.go:356] "Generic (PLEG): container finished" podID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerID="19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a" exitCode=0 Jul 24 23:42:47.425540 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.425400 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" Jul 24 23:42:47.425540 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.425424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" event={"ID":"e9138431-8799-4d72-bf3f-a50061cb54c1","Type":"ContainerDied","Data":"19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a"} Jul 24 23:42:47.425540 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.425477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d" event={"ID":"e9138431-8799-4d72-bf3f-a50061cb54c1","Type":"ContainerDied","Data":"3fb37da7e5d9105af871dd9d6cb280fdbc48a5537c44859ce12976c3c15a7524"} Jul 24 23:42:47.425540 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.425500 2562 scope.go:117] "RemoveContainer" containerID="19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a" Jul 24 23:42:47.433820 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.433804 2562 scope.go:117] "RemoveContainer" containerID="9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e" Jul 24 23:42:47.442694 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.442589 2562 scope.go:117] "RemoveContainer" containerID="19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a" Jul 24 23:42:47.442905 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:47.442880 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a\": container with ID starting with 19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a not found: ID does not exist" containerID="19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a" Jul 24 23:42:47.442967 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.442919 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a"} err="failed to get container status \"19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a\": rpc error: code = NotFound desc = could not find container \"19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a\": container with ID starting with 19e35cbe17e08d241b47c8d865c39a0f0d1470503fb013b7f843d13714b04f3a not found: ID does not exist" Jul 24 23:42:47.442967 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.442942 2562 scope.go:117] "RemoveContainer" containerID="9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e" Jul 24 23:42:47.443222 ip-10-0-143-241 kubenswrapper[2562]: E0724 23:42:47.443202 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e\": container with ID starting with 9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e not found: ID does not exist" containerID="9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e" Jul 24 23:42:47.443270 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.443228 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e"} err="failed to get container status \"9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e\": rpc error: code = NotFound desc = could not find container \"9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e\": container with ID starting with 9e743f0c18439ee2b002af8612f64b4c878a5f8027c4c6fa1a18dd256470f80e not found: ID does not exist" Jul 24 23:42:47.444025 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.444008 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d"] Jul 24 23:42:47.449107 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.449086 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6f3ce2-openshift-default-6f675b4dfb-pw58d"] Jul 24 23:42:47.773546 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:47.773506 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57dbb6486d-xv9p8_4115be71-ee96-45b7-9278-58cf51bef9c9/manager/0.log" Jul 24 23:42:48.131539 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:48.131513 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-777c88b5cf-2ss6p_c5c955d8-5c8a-42bf-baad-bfad7ef79702/manager/0.log" Jul 24 23:42:48.140476 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:48.140452 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-777c88b5cf-2ss6p_c5c955d8-5c8a-42bf-baad-bfad7ef79702/copy-manifests/0.log" Jul 24 23:42:48.851176 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:48.851151 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748915-nbhr8_2d8a366d-f5a5-4437-8939-496a1a1d2bd7/cleanup/0.log" Jul 24 23:42:49.299538 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:49.299449 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" path="/var/lib/kubelet/pods/e9138431-8799-4d72-bf3f-a50061cb54c1/volumes" Jul 24 23:42:51.702086 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:51.702053 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sk55w_93d50af0-a81a-4a54-b190-797c4c83d412/discovery/0.log" Jul 24 23:42:52.057848 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:52.057757 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/1.log" Jul 24 23:42:52.297083 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:52.297052 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ndrmz_d4090740-da29-4dad-b5b7-8a587436717d/payload-processing/0.log" Jul 24 23:42:52.415602 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:52.415567 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-785457d5c7-s7p6m_5b272d32-e8bf-4812-b235-e425e33c7e06/router/0.log" Jul 24 23:42:52.752756 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:52.752666 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-srh9r_5ffad835-f0f2-465e-ab4d-07e4d6037ba5/main/0.log" Jul 24 23:42:53.094450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:42:53.094415 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fbb9lp_c106af9b-3830-4b25-a32e-2aa2f23c6b5b/main/0.log" Jul 24 23:43:01.655156 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:01.655123 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g75zv_d3e447b3-e0fe-4133-a696-e7d1c7f7358e/global-pull-secret-syncer/0.log" Jul 24 23:43:01.753782 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:01.753750 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fz57w_8be5c3d1-c3f6-4834-9a39-e5091bd2f5c0/konnectivity-agent/0.log" Jul 24 23:43:01.849987 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:01.849956 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-241.ec2.internal_b8c2bd3954b3c3942399264f92d6d2f7/haproxy/0.log" Jul 24 23:43:09.858398 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:09.858370 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qjjq4_e100cbed-1739-4a89-85b6-4b7ff1daef3f/node-exporter/0.log" Jul 24 23:43:09.879261 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:09.879233 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qjjq4_e100cbed-1739-4a89-85b6-4b7ff1daef3f/kube-rbac-proxy/0.log" Jul 24 23:43:09.906780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:09.906752 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qjjq4_e100cbed-1739-4a89-85b6-4b7ff1daef3f/init-textfile/0.log" Jul 24 23:43:11.794160 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794126 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t"] Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794422 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787f9e86-38c1-432e-9f5a-99ff55677c50" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794435 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="787f9e86-38c1-432e-9f5a-99ff55677c50" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794442 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794448 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794457 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794462 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794470 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794475 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794480 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794486 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794493 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275642ef-82c7-43fc-9ac6-fc6f70a9dab4" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794498 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="275642ef-82c7-43fc-9ac6-fc6f70a9dab4" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794508 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794512 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794519 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02db8e2-f411-4c01-a230-fbb72ff0d209" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794524 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02db8e2-f411-4c01-a230-fbb72ff0d209" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794531 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794536 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794542 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794547 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794553 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87faf705-25b0-4afe-a412-9ce9958dd0af" containerName="payload-pre-processing" Jul 24 23:43:11.794564 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794558 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="87faf705-25b0-4afe-a412-9ce9958dd0af" containerName="payload-pre-processing" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794608 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794615 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794620 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02db8e2-f411-4c01-a230-fbb72ff0d209" containerName="payload-pre-processing" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794627 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78faec82-39d5-478b-bb74-31f2f191e523" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794633 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794638 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="787f9e86-38c1-432e-9f5a-99ff55677c50" containerName="payload-pre-processing" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794644 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="87faf705-25b0-4afe-a412-9ce9958dd0af" containerName="payload-pre-processing" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794649 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794653 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9138431-8799-4d72-bf3f-a50061cb54c1" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794657 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794662 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5d3251-21d6-4d00-9309-3907a1cadc0f" containerName="istio-proxy" Jul 24 23:43:11.795255 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.794667 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="275642ef-82c7-43fc-9ac6-fc6f70a9dab4" containerName="payload-pre-processing" Jul 24 23:43:11.797840 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.797812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:11.800791 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.800765 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpvfn\"/\"openshift-service-ca.crt\"" Jul 24 23:43:11.802059 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.801701 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpvfn\"/\"kube-root-ca.crt\"" Jul 24 23:43:11.802209 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.802083 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jpvfn\"/\"default-dockercfg-br4hs\"" Jul 24 23:43:11.809325 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.809294 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t"] Jul 24 23:43:11.913836 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.913789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-sys\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:11.914037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.913849 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-proc\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:11.914037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.913936 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-lib-modules\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:11.914037 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.913975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-podres\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:11.914183 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:11.914062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrw6\" (UniqueName: \"kubernetes.io/projected/b062a86e-91de-4d93-ae3a-884f30253da8-kube-api-access-nwrw6\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015328 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-podres\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015536 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrw6\" (UniqueName: \"kubernetes.io/projected/b062a86e-91de-4d93-ae3a-884f30253da8-kube-api-access-nwrw6\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015536 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-sys\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015536 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015417 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-proc\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015536 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-podres\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015536 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-sys\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-lib-modules\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015553 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-proc\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.015780 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.015670 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b062a86e-91de-4d93-ae3a-884f30253da8-lib-modules\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.023881 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.023851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrw6\" (UniqueName: \"kubernetes.io/projected/b062a86e-91de-4d93-ae3a-884f30253da8-kube-api-access-nwrw6\") pod \"perf-node-gather-daemonset-z2g5t\" (UID: \"b062a86e-91de-4d93-ae3a-884f30253da8\") " pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.112863 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.112830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.248453 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.248324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t"] Jul 24 23:43:12.251189 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:43:12.251163 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb062a86e_91de_4d93_ae3a_884f30253da8.slice/crio-0d0b9bca44dd1a5ffcdbcf24cf5db90ff355af949d1c116af91109431118dd65 WatchSource:0}: Error finding container 0d0b9bca44dd1a5ffcdbcf24cf5db90ff355af949d1c116af91109431118dd65: Status 404 returned error can't find the container with id 0d0b9bca44dd1a5ffcdbcf24cf5db90ff355af949d1c116af91109431118dd65 Jul 24 23:43:12.535950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.535857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" event={"ID":"b062a86e-91de-4d93-ae3a-884f30253da8","Type":"ContainerStarted","Data":"b45817539f6db296db632d4816317682c5b9561a4f5864cf4131e5debda48649"} Jul 24 23:43:12.535950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.535912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" event={"ID":"b062a86e-91de-4d93-ae3a-884f30253da8","Type":"ContainerStarted","Data":"0d0b9bca44dd1a5ffcdbcf24cf5db90ff355af949d1c116af91109431118dd65"} Jul 24 23:43:12.536144 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.535963 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:12.570835 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:12.570784 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" podStartSLOduration=1.570766052 podStartE2EDuration="1.570766052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:43:12.56907336 +0000 UTC m=+3051.844042466" watchObservedRunningTime="2026-07-24 23:43:12.570766052 +0000 UTC m=+3051.845735147" Jul 24 23:43:13.631964 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:13.631937 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-wqjpr_71ccb923-5f3c-4067-b7e2-e898e4841e81/volume-data-source-validator/0.log" Jul 24 23:43:14.768700 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:14.768643 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sq92q_e800f15f-9f18-4c15-8baa-63cfaaee1138/dns/0.log" Jul 24 23:43:14.788966 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:14.788938 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sq92q_e800f15f-9f18-4c15-8baa-63cfaaee1138/kube-rbac-proxy/0.log" Jul 24 23:43:14.836662 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:14.836630 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6dnt_ce701a92-c7b7-43bf-8661-d2d5078f4836/dns-node-resolver/0.log" Jul 24 23:43:15.442210 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:15.442168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-24kn8_a69aefd5-23e0-45b8-b9d0-a87a4e2aabe6/node-ca/0.log" Jul 24 23:43:16.683430 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:16.683393 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sk55w_93d50af0-a81a-4a54-b190-797c4c83d412/discovery/0.log" Jul 24 23:43:16.761343 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:16.761306 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/0.log" Jul 24 23:43:16.816128 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:16.816095 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/1.log" Jul 24 23:43:16.887889 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:16.887854 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ndrmz_d4090740-da29-4dad-b5b7-8a587436717d/payload-processing/0.log" Jul 24 23:43:16.911391 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:16.911365 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-785457d5c7-s7p6m_5b272d32-e8bf-4812-b235-e425e33c7e06/router/0.log" Jul 24 23:43:17.737284 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:17.737254 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4hwnx_c265bfe7-129d-4723-8348-3c9890868872/serve-healthcheck-canary/0.log" Jul 24 23:43:18.278406 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:18.278369 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dncjt_22d61829-69c3-4f6e-9c91-26c8ac70b188/kube-rbac-proxy/0.log" Jul 24 23:43:18.299892 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:18.299863 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dncjt_22d61829-69c3-4f6e-9c91-26c8ac70b188/exporter/0.log" Jul 24 23:43:18.326247 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:18.326164 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dncjt_22d61829-69c3-4f6e-9c91-26c8ac70b188/extractor/0.log" Jul 24 23:43:18.549013 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:18.548927 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jpvfn/perf-node-gather-daemonset-z2g5t" Jul 24 23:43:21.460359 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:21.460324 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57dbb6486d-xv9p8_4115be71-ee96-45b7-9278-58cf51bef9c9/manager/0.log" Jul 24 23:43:21.710853 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:21.710764 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-777c88b5cf-2ss6p_c5c955d8-5c8a-42bf-baad-bfad7ef79702/manager/0.log" Jul 24 23:43:21.733376 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:21.733343 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-777c88b5cf-2ss6p_c5c955d8-5c8a-42bf-baad-bfad7ef79702/copy-manifests/0.log" Jul 24 23:43:23.109521 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.109486 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bccc7ffcc-spfnk_5cd3fff4-6905-45b7-b74d-c71a5cfa54db/manager/0.log" Jul 24 23:43:23.136932 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.136899 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nbk7s_27aba347-8a24-4cd1-a1dc-1d6b909944a1/openshift-lws-operator/0.log" Jul 24 23:43:23.152541 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.152508 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sk55w_93d50af0-a81a-4a54-b190-797c4c83d412/discovery/0.log" Jul 24 23:43:23.210450 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.210416 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/0.log" Jul 24 23:43:23.268581 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.268549 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/1.log" Jul 24 23:43:23.319442 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.319403 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ndrmz_d4090740-da29-4dad-b5b7-8a587436717d/payload-processing/0.log" Jul 24 23:43:23.334171 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:23.334144 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-785457d5c7-s7p6m_5b272d32-e8bf-4812-b235-e425e33c7e06/router/0.log" Jul 24 23:43:24.053360 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.053326 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2jjgl"] Jul 24 23:43:24.053648 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.053636 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" Jul 24 23:43:24.053648 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.053650 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbd8504-5cfd-4adb-8479-a73832b79f0f" containerName="istio-proxy" Jul 24 23:43:24.059812 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.059781 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" Jul 24 23:43:24.062702 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.062659 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-5dg96\"" Jul 24 23:43:24.063902 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.063882 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 23:43:24.064022 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.063913 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 23:43:24.067949 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.067923 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2jjgl"] Jul 24 23:43:24.221426 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.221381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtd5\" (UniqueName: \"kubernetes.io/projected/a8b76d25-97e4-4836-ad9d-7f0ae6854e8d-kube-api-access-7qtd5\") pod \"kuadrant-operator-catalog-2jjgl\" (UID: \"a8b76d25-97e4-4836-ad9d-7f0ae6854e8d\") " pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" Jul 24 23:43:24.322405 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.322305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtd5\" (UniqueName: \"kubernetes.io/projected/a8b76d25-97e4-4836-ad9d-7f0ae6854e8d-kube-api-access-7qtd5\") pod \"kuadrant-operator-catalog-2jjgl\" (UID: \"a8b76d25-97e4-4836-ad9d-7f0ae6854e8d\") " pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" Jul 24 23:43:24.333138 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.333105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtd5\" (UniqueName: \"kubernetes.io/projected/a8b76d25-97e4-4836-ad9d-7f0ae6854e8d-kube-api-access-7qtd5\") pod \"kuadrant-operator-catalog-2jjgl\" (UID: \"a8b76d25-97e4-4836-ad9d-7f0ae6854e8d\") " pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" Jul 24 23:43:24.370166 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.370129 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" Jul 24 23:43:24.506971 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.506941 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2jjgl"] Jul 24 23:43:24.509116 ip-10-0-143-241 kubenswrapper[2562]: W0724 23:43:24.509088 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b76d25_97e4_4836_ad9d_7f0ae6854e8d.slice/crio-327d0bdbaa4b325ec2195b1b88915fb25546b3c068320219b855b1e0bcc03513 WatchSource:0}: Error finding container 327d0bdbaa4b325ec2195b1b88915fb25546b3c068320219b855b1e0bcc03513: Status 404 returned error can't find the container with id 327d0bdbaa4b325ec2195b1b88915fb25546b3c068320219b855b1e0bcc03513 Jul 24 23:43:24.510950 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.510931 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:43:24.589949 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.589905 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" event={"ID":"a8b76d25-97e4-4836-ad9d-7f0ae6854e8d","Type":"ContainerStarted","Data":"327d0bdbaa4b325ec2195b1b88915fb25546b3c068320219b855b1e0bcc03513"} Jul 24 23:43:24.968414 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:24.968322 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-sk55w_93d50af0-a81a-4a54-b190-797c4c83d412/discovery/0.log" Jul 24 23:43:25.018973 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:25.018933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/0.log" Jul 24 23:43:25.082721 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:25.082692 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4fpq7_eb1c31c7-d8f4-452f-b987-7ecbddadb567/istio-proxy/1.log" Jul 24 23:43:25.146194 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:25.146166 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-ndrmz_d4090740-da29-4dad-b5b7-8a587436717d/payload-processing/0.log" Jul 24 23:43:25.163757 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:25.163728 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-785457d5c7-s7p6m_5b272d32-e8bf-4812-b235-e425e33c7e06/router/0.log" Jul 24 23:43:27.609428 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:27.609383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" event={"ID":"a8b76d25-97e4-4836-ad9d-7f0ae6854e8d","Type":"ContainerStarted","Data":"7bfc0b574fa7397ed29e248f5f46c98ae431b8e67fe1aec1fd2b351e6a073f7e"} Jul 24 23:43:27.626175 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:27.626121 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2jjgl" podStartSLOduration=1.5938395509999999 podStartE2EDuration="3.626103543s" totalImagesPullingTime="2.032263992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:43:27.624723721 +0000 UTC m=+3066.899692815" watchObservedRunningTime="2026-07-24 23:43:27.626103543 +0000 UTC m=+3066.901072637" Jul 24 23:43:28.040875 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:28.040796 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-j7lzw_042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a/migrator/0.log" Jul 24 23:43:28.063630 ip-10-0-143-241 kubenswrapper[2562]: I0724 23:43:28.063598 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-j7lzw_042186f1-d73b-4ba2-8c3e-b9ff3dbfd37a/graceful-termination/0.log"